![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3626485%2F3ecc17ccdeead41cba76a26a6087d03b%2FGraphReductions3.png?generation=1700281118388593&alt=media\")
The models all broadly followed the following format:
\nThe Tile Dataset result was a single model following this design, with 3 GraphSAGE layers and 3 Linear Layers trained with ListMLE loss. The Layout Dataset results were taken from an ensemble of models with slight variations in their design. All models used GeLu activations, but differed in other respects (detailed below). Output losses used were ListMLE and Pairwise Hinge.
\nWe kept the same Train/Val split as provided in the competition dataset.
\nEach layout graph was transformed into two distinct graphs:
\n
Before even talking about new features, it's worth mentioning that normalization is absolutely essential on this problem. If you didn't normalize your data, you didn't score well.
\nWe defined a few extra features. For the nodes, we defined:
\nAdditional config features were computed for the output, input and kernel sections. Each feature was replicated for each of those sections:
\nAdditionally, we computed similarity metrics for
\nOpcodes were just one-hot encoded.
\nThe is_default flag was introduced because it seemed like the random vs. default distributions were different enough to warrant having predictive value, since the test set also contained this information. It seemed to provide a small but reliable boost to val scores.
\nFor the layout problem, we defined the following GraphBlock, using a GAT with (out channels / 2) channels to process the node features given the 3-Hop Graph, and GAT or GraphSAGE with (out channels / 2) channels to process the features given the Config Positioning graph.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3626485%2F893f9ce0ec19a542d2156afc8e951c66%2FGraphBlock2.png?generation=1700278642575220&alt=media\")
We then layered the GraphBlocks with residual connections and added some dense feed-forward layers (also with residuals) to which we concatenated the global features. The final result was an ensemble of slightly different versions of this model (varying hidden dims, linear layers, etc.).
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3626485%2Fc9cfbe2d9197c006df390155fbaab177%2FModelDiagram.png?generation=1700279330883081&alt=media\")
The code for of multi-edge block used in the Layout Set is shown below (the Tile Set model is very similar, with fewer complicated pieces. It's less polished but you can view it here. The rest of the code is available here
\n (nn.Module):\n ():\n \n\n ().__init__()\n\n output_dim_per_block = output_dim // \n\n main_block == :\n self.main_edge_block = GATBlock(\n input_dim,\n output_dim_per_block,\n heads=heads,\n with_residual=,\n dropout=dropout,\n )\n :\n self.main_edge_block = SAGEBlock(\n input_dim,\n output_dim_per_block,\n with_residual=,\n dropout=dropout,\n )\n\n alt_block == :\n self.alternate_edge_block = GATBlock(\n input_dim,\n output_dim_per_block,\n heads=heads,\n with_residual=,\n dropout=dropout,\n )\n\n :\n self.alternate_edge_block = SAGEBlock(\n input_dim,\n output_dim_per_block,\n with_residual=,\n dropout=dropout,\n )\n\n self.with_residual = with_residual\n self.output_dim = output_dim\n\n ():\n main_edge_index = data.edge_index\n alternate_edge_index = data.alt_edge_index\n\n main_edge_data = Data(\n x=data.x,\n edge_index=main_edge_index,\n batch=data.batch,\n )\n\n alternate_edge_data = Data(\n x=data.x,\n edge_index=alternate_edge_index,\n batch=data.batch,\n )\n\n main_edge_data = self.main_edge_block(main_edge_data)\n alternate_edge_data = self.alternate_edge_block(alternate_edge_data)\n\n f = torch.cat([main_edge_data.x, alternate_edge_data.x], dim=)\n\n self.with_residual:\n f += data.x\n\n new_data = Data(\n x=f,\n batch=data.batch,\n )\n\n data.update(new_data)\nThe Model for the Tile Dataset is more or less exactly the same, except it used 3 graph layers, 3 linear layers, and no graph reduction at all (because there were no configurable node).
\nNLP and XLA were trained separately. I would have loved to play with a unified model more, but I ran out of time and compute, and they seemed to learn well when they were separate.
\nAll models used:
\nOther parameters are as follows:
\n| Parameter | \nXLA-1 | \nXLA-2 | \nNLP-1 | \nNLP-2 | \n
|---|---|---|---|---|
| Loss | \nlistMLE | \nlistMLE | \nlistMLE | \nRank Margin Loss | \n
| Learning Rate | \n0.00028 | \n0.00028 | \n0.00028 | \n0.0001 | \n
| Weight Decay | \n0.004 | \n0.004 | \n0.004 | \n0.007 | \n
| Graph Layers | \n4 | \n4 | \n4 | \n4 | \n
| Graph Channels | \n128 | \n128 | \n128 | \n128 | \n
| FF Layers | \n1 | \n2 | \n1 | \n3 | \n
| FF Channels | \n128 | \n128 | \n128 | \n64 | \n
| 3-Hop Graph Conv | \nGAT(heads=8) | \nGAT(heads=8) | \nGAT(heads=8) | \nGAT(heads=1) | \n
| Config Graph Conv | \nGAT | \nGAT | \nGAT | \nGraphSAGE | \n
| Dropout | \n0.15 | \n0.15 | \n0 | \n0 | \n
Outputs were collected from XLA-1 after 3 epochs, and 2 snapshots of XLA-2 in training (end of epochs 2 and 3) based on their val scores.
\nThe NLP Models never even finished one epoch, the loss appeared to plateau and I only had so much compute. This indicates to me that I probably could have tuned the LR better or regularized better.
\nFor a given file id (e.g \"xla:default:abc…\"), we have N (1000 or 1001) predictions per model output. We min-max normalize the predictions so they're all in a zero-to-one range, then we just add the scores elementwise for each model output. We use the summed config scores to derive the rank ordering.
\nThe normalization is important here, because the models are not guaranteed to be outputting numbers on the same scale if you're using a ranking loss.
\nI also tried simple rank averaging and Borda count, both of which worked but not as well as the min-max averaging. This is likely because these methods can't account for things like \"how much better is rank 2 than rank 3\", while the min-max normalized ensemble can.
\nThe models all broadly followed the following format:
\nThe Tile Dataset result was a single model following this design, with 3 GraphSAGE layers and 3 Linear Layers trained with ListMLE loss. The Layout Dataset results were taken from an ensemble of models with slight variations in their design. All models used GeLu activations, but differed in other respects (detailed below). Output losses used were ListMLE and Pairwise Hinge.
\nWe kept the same Train/Val split as provided in the competition dataset.
\nEach layout graph was transformed into two distinct graphs:
\n
Before even talking about new features, it's worth mentioning that normalization is absolutely essential on this problem. If you didn't normalize your data, you didn't score well.
\nWe defined a few extra features. For the nodes, we defined:
\nAdditional config features were computed for the output, input and kernel sections. Each feature was replicated for each of those sections:
\nAdditionally, we computed similarity metrics for
\nOpcodes were just one-hot encoded.
\nThe is_default flag was introduced because it seemed like the random vs. default distributions were different enough to warrant having predictive value, since the test set also contained this information. It seemed to provide a small but reliable boost to val scores.
\nFor the layout problem, we defined the following GraphBlock, using a GAT with (out channels / 2) channels to process the node features given the 3-Hop Graph, and GAT or GraphSAGE with (out channels / 2) channels to process the features given the Config Positioning graph.
\n
We then layered the GraphBlocks with residual connections and added some dense feed-forward layers (also with residuals) to which we concatenated the global features. The final result was an ensemble of slightly different versions of this model (varying hidden dims, linear layers, etc.).
\n
The code for of multi-edge block used in the Layout Set is shown below (the Tile Set model is very similar, with fewer complicated pieces. It's less polished but you can view it here. The rest of the code is available here
\n (nn.Module):\n ():\n \n\n ().__init__()\n\n output_dim_per_block = output_dim // \n\n main_block == :\n self.main_edge_block = GATBlock(\n input_dim,\n output_dim_per_block,\n heads=heads,\n with_residual=,\n dropout=dropout,\n )\n :\n self.main_edge_block = SAGEBlock(\n input_dim,\n output_dim_per_block,\n with_residual=,\n dropout=dropout,\n )\n\n alt_block == :\n self.alternate_edge_block = GATBlock(\n input_dim,\n output_dim_per_block,\n heads=heads,\n with_residual=,\n dropout=dropout,\n )\n\n :\n self.alternate_edge_block = SAGEBlock(\n input_dim,\n output_dim_per_block,\n with_residual=,\n dropout=dropout,\n )\n\n self.with_residual = with_residual\n self.output_dim = output_dim\n\n ():\n main_edge_index = data.edge_index\n alternate_edge_index = data.alt_edge_index\n\n main_edge_data = Data(\n x=data.x,\n edge_index=main_edge_index,\n batch=data.batch,\n )\n\n alternate_edge_data = Data(\n x=data.x,\n edge_index=alternate_edge_index,\n batch=data.batch,\n )\n\n main_edge_data = self.main_edge_block(main_edge_data)\n alternate_edge_data = self.alternate_edge_block(alternate_edge_data)\n\n f = torch.cat([main_edge_data.x, alternate_edge_data.x], dim=)\n\n self.with_residual:\n f += data.x\n\n new_data = Data(\n x=f,\n batch=data.batch,\n )\n\n data.update(new_data)\nThe Model for the Tile Dataset is more or less exactly the same, except it used 3 graph layers, 3 linear layers, and no graph reduction at all (because there were no configurable node).
\nNLP and XLA were trained separately. I would have loved to play with a unified model more, but I ran out of time and compute, and they seemed to learn well when they were separate.
\nAll models used:
\nOther parameters are as follows:
\n| Parameter | \nXLA-1 | \nXLA-2 | \nNLP-1 | \nNLP-2 | \n
|---|---|---|---|---|
| Loss | \nlistMLE | \nlistMLE | \nlistMLE | \nRank Margin Loss | \n
| Learning Rate | \n0.00028 | \n0.00028 | \n0.00028 | \n0.0001 | \n
| Weight Decay | \n0.004 | \n0.004 | \n0.004 | \n0.007 | \n
| Graph Layers | \n4 | \n4 | \n4 | \n4 | \n
| Graph Channels | \n128 | \n128 | \n128 | \n128 | \n
| FF Layers | \n1 | \n2 | \n1 | \n3 | \n
| FF Channels | \n128 | \n128 | \n128 | \n64 | \n
| 3-Hop Graph Conv | \nGAT(heads=8) | \nGAT(heads=8) | \nGAT(heads=8) | \nGAT(heads=1) | \n
| Config Graph Conv | \nGAT | \nGAT | \nGAT | \nGraphSAGE | \n
| Dropout | \n0.15 | \n0.15 | \n0 | \n0 | \n
Outputs were collected from XLA-1 after 3 epochs, and 2 snapshots of XLA-2 in training (end of epochs 2 and 3) based on their val scores.
\nThe NLP Models never even finished one epoch, the loss appeared to plateau and I only had so much compute. This indicates to me that I probably could have tuned the LR better or regularized better.
\nFor a given file id (e.g \"xla:default:abc…\"), we have N (1000 or 1001) predictions per model output. We min-max normalize the predictions so they're all in a zero-to-one range, then we just add the scores elementwise for each model output. We use the summed config scores to derive the rank ordering.
\nThe normalization is important here, because the models are not guaranteed to be outputting numbers on the same scale if you're using a ranking loss.
\nI also tried simple rank averaging and Borda count, both of which worked but not as well as the min-max averaging. This is likely because these methods can't account for things like \"how much better is rank 2 than rank 3\", while the min-max normalized ensemble can.
\n