{
  "id": 456489,
  "title": "16th place solution",
  "url": "/competitions/predict-ai-model-runtime/discussion/456489",
  "author_name": "Yuki Okumura",
  "post_date": "2023-11-20T09:43:53.857000",
  "votes": 7,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Thanks for hosting the competition on a very interesting topic and congratulations to all the winners!</p>\n<p>I'll share my solution briefly.</p>\n<h1>Summary</h1>\n<ul>\n<li>3-hop subgraphs from configurable nodes</li>\n<li>Drop duplicated configs</li>\n<li>Listwise loss. In my experiments, listwise loss converged faster and performed better than pairwise loss.</li>\n<li>3-layer <code>SAGEConv</code> with <code>LayerNorm</code> and residual connections. I implemented residual connections by simply adding the initial embedding to the output of each layer. Below is a code snippet.</li>\n</ul>\n<pre><code> ():\n    node_opcode = batch.node_opcode.long()\n\n    opcode_embeds = self.opcode_embedding(node_opcode)\n\n    x = torch.concat([batch.node_feat, opcode_embeds, batch.node_config_feat * self.node_config_weights], dim=)\n    x = self.lin1(x)\n    x = self.norm1(x).relu()\n\n    x_init = x\n     i  (self.n_layers):\n        x = self.convs[i](x, batch.edge_index)\n        x = self.norms[i](x).relu()\n        x = x_init + x\n\n    x = torch.concat([global_mean_pool(x, batch.batch), global_max_pool(x, batch.batch)], dim=)\n    x = self.dropout(x)\n    x = self.readout(x)\n\n     x\n</code></pre>\n<ul>\n<li><p>Models were trained separately for diffrent subtypes</p></li>\n<li><p>CV scores (provided&nbsp;train,&nbsp;valid splits)</p>\n<ul>\n<li>xla default: 0.37</li>\n<li>xla random: 0.71</li>\n<li>nlp default: 0.55</li>\n<li>nlp random: 0.96</li>\n<li>tile: 0.97</li></ul>\n<p>With these CV scores, I got a score of 0.684 (public) and 0.688 (private). </p></li>\n</ul>",
  "messages": [
    {
      "id": 2531532,
      "postDate": "2023-11-20T09:43:53.857Z",
      "content": "<p>Thanks for hosting the competition on a very interesting topic and congratulations to all the winners!</p>\n<p>I'll share my solution briefly.</p>\n<h1>Summary</h1>\n<ul>\n<li>3-hop subgraphs from configurable nodes</li>\n<li>Drop duplicated configs</li>\n<li>Listwise loss. In my experiments, listwise loss converged faster and performed better than pairwise loss.</li>\n<li>3-layer <code>SAGEConv</code> with <code>LayerNorm</code> and residual connections. I implemented residual connections by simply adding the initial embedding to the output of each layer. Below is a code snippet.</li>\n</ul>\n<pre><code> ():\n    node_opcode = batch.node_opcode.long()\n\n    opcode_embeds = self.opcode_embedding(node_opcode)\n\n    x = torch.concat([batch.node_feat, opcode_embeds, batch.node_config_feat * self.node_config_weights], dim=)\n    x = self.lin1(x)\n    x = self.norm1(x).relu()\n\n    x_init = x\n     i  (self.n_layers):\n        x = self.convs[i](x, batch.edge_index)\n        x = self.norms[i](x).relu()\n        x = x_init + x\n\n    x = torch.concat([global_mean_pool(x, batch.batch), global_max_pool(x, batch.batch)], dim=)\n    x = self.dropout(x)\n    x = self.readout(x)\n\n     x\n</code></pre>\n<ul>\n<li><p>Models were trained separately for diffrent subtypes</p></li>\n<li><p>CV scores (provided&nbsp;train,&nbsp;valid splits)</p>\n<ul>\n<li>xla default: 0.37</li>\n<li>xla random: 0.71</li>\n<li>nlp default: 0.55</li>\n<li>nlp random: 0.96</li>\n<li>tile: 0.97</li></ul>\n<p>With these CV scores, I got a score of 0.684 (public) and 0.688 (private). </p></li>\n</ul>",
      "rawMarkdown": "Thanks for hosting the competition on a very interesting topic and congratulations to all the winners!\n\nI'll share my solution briefly.\n\n# Summary\n\n- 3-hop subgraphs from configurable nodes\n- Drop duplicated configs\n- Listwise loss. In my experiments, listwise loss converged faster and performed better than pairwise loss.\n- 3-layer `SAGEConv` with `LayerNorm` and residual connections. I implemented residual connections by simply adding the initial embedding to the output of each layer. Below is a code snippet.\n\n```python\ndef forward(self, batch):\n    node_opcode = batch.node_opcode.long()\n\n    opcode_embeds = self.opcode_embedding(node_opcode)\n\n    x = torch.concat([batch.node_feat, opcode_embeds, batch.node_config_feat * self.node_config_weights], dim=1)\n    x = self.lin1(x)\n    x = self.norm1(x).relu()\n\n    x_init = x\n    for i in range(self.n_layers):\n        x = self.convs[i](x, batch.edge_index)\n        x = self.norms[i](x).relu()\n        x = x_init + x\n\n    x = torch.concat([global_mean_pool(x, batch.batch), global_max_pool(x, batch.batch)], dim=1)\n    x = self.dropout(x)\n    x = self.readout(x)\n\n    return x\n```\n\n- Models were trained separately for diffrent subtypes\n- CV scores (provided train, valid splits)\n    - xla default: 0.37\n    - xla random: 0.71\n    - nlp default: 0.55\n    - nlp random: 0.96\n    - tile: 0.97\n    \n    With these CV scores, I got a score of 0.684 (public) and 0.688 (private). \n    ",
      "votes": 7
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2531532": "Thanks for hosting the competition on a very interesting topic and congratulations to all the winners!\n\nI'll share my solution briefly.\n\n# Summary\n\n- 3-hop subgraphs from configurable nodes\n- Drop duplicated configs\n- Listwise loss. In my experiments, listwise loss converged faster and performed better than pairwise loss.\n- 3-layer `SAGEConv` with `LayerNorm` and residual connections. I implemented residual connections by simply adding the initial embedding to the output of each layer. Below is a code snippet.\n\n```python\ndef forward(self, batch):\n    node_opcode = batch.node_opcode.long()\n\n    opcode_embeds = self.opcode_embedding(node_opcode)\n\n    x = torch.concat([batch.node_feat, opcode_embeds, batch.node_config_feat * self.node_config_weights], dim=1)\n    x = self.lin1(x)\n    x = self.norm1(x).relu()\n\n    x_init = x\n    for i in range(self.n_layers):\n        x = self.convs[i](x, batch.edge_index)\n        x = self.norms[i](x).relu()\n        x = x_init + x\n\n    x = torch.concat([global_mean_pool(x, batch.batch), global_max_pool(x, batch.batch)], dim=1)\n    x = self.dropout(x)\n    x = self.readout(x)\n\n    return x\n```\n\n- Models were trained separately for diffrent subtypes\n- CV scores (provided train, valid splits)\n    - xla default: 0.37\n    - xla random: 0.71\n    - nlp default: 0.55\n    - nlp random: 0.96\n    - tile: 0.97\n    \n    With these CV scores, I got a score of 0.684 (public) and 0.688 (private). \n    "
  }
}