{ "id": 456462, "title": "4th simple mlp solution", "url": "/competitions/predict-ai-model-runtime/writeups/ri-ethan-4th-simple-mlp-solution", "author_name": "", "post_date": "2023-11-20T07:07:42.147Z", "votes": 25, "comment_count": 6, "views": 0, "content": "

`class SimpleMLP(torch.nn.Module):

\n
def (self):\n    ().()\n    self.embedding = torch.nn.(,)\n    self.config_dense = torch.nn.(nn.(+, ),\n                                            nn.(),\n                                            nn.(, ),\n                                            nn.(),\n                                    )\n    self.node_dense = torch.nn.(nn.(*+, ),\n                                          nn.(),\n                                          nn.(, ),\n                                          nn.(),\n                                    )\n    self.output=torch.nn.(nn.(*+, ),\n                                    nn.(),\n                                    nn.(, ),\n                                    )\ndef (self,x_cfg,x_feat,x_op):\n    x_feat=x_feat.()\n    x_op=x_op.()\n    x_op = self.(x_op).(-,)\n    x_feat = torch.([x_feat,x_op],dim =)\n    x_feat = self.(x_feat)\n    x_graph = x_feat.()\n    x_graph = x_graph.((x_cfg),dim=)\n    x_cfg = torch.([x_cfg,x_graph],axis=)\n    x_cfg = self.(x_cfg)\n    x=(x_feat.T@x_cfg).((x_cfg),-)\n    x_cfg_mean=x_cfg.(dim=)\n    x = torch.([x,x_cfg_mean],axis=)\n    x=self.(x)\n    x=torch.(x)\n    return x\n
\n

def load_data(row):

\n
node_feat_index=[,,,,,\n             ,,,,\n             ,,,\n            ]\nconfig_index=[,,,,,,,]\ndata= dict(np.load(row.path))\n=data[][:,node_feat_index]\nnode_feat=pd.(,columns=[\nnode_feat[]=range(len(node_feat))\nnode_feat_link=node_feat.copy()\nnode_feat_link.columns=[i+ for i in node_feat_link.columns]\nnode_config_ids=pd.({:data[],:range(len(data[]))})\nnode_opcode=pd.({:range(len(data[])),:data[]})\nedge_index=pd.(data[],columns=[,])\nedge_index=edge_index[edge_index.id.isin(data[])]\nedge_index=edge_index.merge(node_opcode,on=,how=).merge(node_opcode.rename(columns={:,:}),on=,how=)\nedge_index=edge_index.merge(node_config_ids,on=,how=)\nedge_index=edge_index.merge(node_feat,on=,how=).merge(node_feat_link,on=,how=)\nall_features=[i for i in edge_index.columns if i not in [, , ,, ]]\nnode_feat_array=edge_index[all_features].values.astype(np.float32)\nnode_config_feat=data[][:,:,config_index]\nnode_opcode_array=edge_index[[,]].values\nnode_config_feat=node_config_feat[:,edge_index[].values,:]\nlabel=data[].argsort().argsort()/len(data[])\nreturn {:node_feat_array,              #x_feat\n        :node_config_feat,      #x_cfg\n        :node_opcode_array,          #x_op\n        :label,\n        :data[],\n        :np.array(range(len(label)))\n       }
", "messages": [ { "id": "2531381", "postDate": "11/20/2023 06:57:01", "content": "

`class SimpleMLP(torch.nn.Module):

\n
def (self):\n    ().()\n    self.embedding = torch.nn.(,)\n    self.config_dense = torch.nn.(nn.(+, ),\n                                            nn.(),\n                                            nn.(, ),\n                                            nn.(),\n                                    )\n    self.node_dense = torch.nn.(nn.(*+, ),\n                                          nn.(),\n                                          nn.(, ),\n                                          nn.(),\n                                    )\n    self.output=torch.nn.(nn.(*+, ),\n                                    nn.(),\n                                    nn.(, ),\n                                    )\ndef (self,x_cfg,x_feat,x_op):\n    x_feat=x_feat.()\n    x_op=x_op.()\n    x_op = self.(x_op).(-,)\n    x_feat = torch.([x_feat,x_op],dim =)\n    x_feat = self.(x_feat)\n    x_graph = x_feat.()\n    x_graph = x_graph.((x_cfg),dim=)\n    x_cfg = torch.([x_cfg,x_graph],axis=)\n    x_cfg = self.(x_cfg)\n    x=(x_feat.T@x_cfg).((x_cfg),-)\n    x_cfg_mean=x_cfg.(dim=)\n    x = torch.([x,x_cfg_mean],axis=)\n    x=self.(x)\n    x=torch.(x)\n    return x\n
\n

def load_data(row):

\n
node_feat_index=[,,,,,\n             ,,,,\n             ,,,\n            ]\nconfig_index=[,,,,,,,]\ndata= dict(np.load(row.path))\n=data[][:,node_feat_index]\nnode_feat=pd.(,columns=[\nnode_feat[]=range(len(node_feat))\nnode_feat_link=node_feat.copy()\nnode_feat_link.columns=[i+ for i in node_feat_link.columns]\nnode_config_ids=pd.({:data[],:range(len(data[]))})\nnode_opcode=pd.({:range(len(data[])),:data[]})\nedge_index=pd.(data[],columns=[,])\nedge_index=edge_index[edge_index.id.isin(data[])]\nedge_index=edge_index.merge(node_opcode,on=,how=).merge(node_opcode.rename(columns={:,:}),on=,how=)\nedge_index=edge_index.merge(node_config_ids,on=,how=)\nedge_index=edge_index.merge(node_feat,on=,how=).merge(node_feat_link,on=,how=)\nall_features=[i for i in edge_index.columns if i not in [, , ,, ]]\nnode_feat_array=edge_index[all_features].values.astype(np.float32)\nnode_config_feat=data[][:,:,config_index]\nnode_opcode_array=edge_index[[,]].values\nnode_config_feat=node_config_feat[:,edge_index[].values,:]\nlabel=data[].argsort().argsort()/len(data[])\nreturn {:node_feat_array,              #x_feat\n        :node_config_feat,      #x_cfg\n        :node_opcode_array,          #x_op\n        :label,\n        :data[],\n        :np.array(range(len(label)))\n       }
", "rawMarkdown": "`class SimpleMLP(torch.nn.Module):\n\n def __init__(self):\n super().__init__()\n self.embedding = torch.nn.Embedding(120,10)\n self.config_dense = torch.nn.Sequential(nn.Linear(128+8, 1024),\n nn.ReLU(),\n nn.Linear(1024, 128),\n nn.ReLU(),\n )\n self.node_dense = torch.nn.Sequential(nn.Linear(12*2+20, 1024),\n nn.ReLU(),\n nn.Linear(1024, 128),\n nn.ReLU(),\n )\n self.output=torch.nn.Sequential(nn.Linear(128*128+128, 1024),\n nn.ReLU(),\n nn.Linear(1024, 1),\n )\n def forward(self,x_cfg,x_feat,x_op):\n x_feat=x_feat.squeeze()\n x_op=x_op.squeeze()\n x_op = self.embedding(x_op).reshape(-1,20)\n x_feat = torch.concat([x_feat,x_op],dim =1)\n x_feat = self.node_dense(x_feat)\n x_graph = x_feat.unsqueeze(0)\n x_graph = x_graph.repeat_interleave(len(x_cfg),dim=0)\n x_cfg = torch.concat([x_cfg,x_graph],axis=2)\n x_cfg = self.config_dense(x_cfg)\n x=(x_feat.T@x_cfg).reshape(len(x_cfg),-1)\n x_cfg_mean=x_cfg.mean(dim=1)\n x = torch.concat([x,x_cfg_mean],axis=1)\n x=self.output(x)\n x=torch.flatten(x)\n return x\n \n\ndef load_data(row):\n\n node_feat_index=[21,22,23,24,28,\n 101,102,103,104,\n 134,135,136,\n ]\n config_index=[0,1,2,6,7,8,12,13]\n data= dict(np.load(row.path))\n X=data[\"node_feat\"][:,node_feat_index]\n node_feat=pd.DataFrame(X,columns=[\"fe%s\"% i for i in range(X.shape[1])])\n node_feat[\"id\"]=range(len(node_feat))\n node_feat_link=node_feat.copy()\n node_feat_link.columns=[i+\"_link\" for i in node_feat_link.columns]\n node_config_ids=pd.DataFrame({\"id\":data[\"node_config_ids\"],\"ind\":range(len(data[\"node_config_ids\"]))})\n node_opcode=pd.DataFrame({\"id\":range(len(data[\"node_opcode\"])),\"node_opcode\":data[\"node_opcode\"]})\n edge_index=pd.DataFrame(data[\"edge_index\"],columns=[\"id\",\"id_link\"])\n edge_index=edge_index[edge_index.id.isin(data[\"node_config_ids\"])]\n edge_index=edge_index.merge(node_opcode,on=\"id\",how=\"left\").merge(node_opcode.rename(columns={\"id\":\"id_link\",\"node_opcode\":\"node_opcode_link\"}),on=\"id_link\",how=\"left\")\n edge_index=edge_index.merge(node_config_ids,on=\"id\",how=\"left\")\n edge_index=edge_index.merge(node_feat,on=\"id\",how=\"left\").merge(node_feat_link,on=\"id_link\",how=\"left\")\n all_features=[i for i in edge_index.columns if i not in ['id', 'id_link', 'node_opcode','node_opcode_link', 'ind']]\n node_feat_array=edge_index[all_features].values.astype(np.float32)\n node_config_feat=data[\"node_config_feat\"][:,:,config_index]\n node_opcode_array=edge_index[[\"node_opcode\",\"node_opcode_link\"]].values\n node_config_feat=node_config_feat[:,edge_index[\"ind\"].values,:]\n label=data[\"config_runtime\"].argsort().argsort()/len(data[\"config_runtime\"])\n return {\"node_feat\":node_feat_array, #x_feat\n \"node_config_feat\":node_config_feat, #x_cfg\n \"node_opcode\":node_opcode_array, #x_op\n \"target\":label,\n \"config_runtime\":data[\"config_runtime\"],\n \"ind\":np.array(range(len(label)))\n }`", "votes": null }, { "id": "2531962", "postDate": "11/20/2023 17:12:06", "content": "

surprise that MLP can work.
\ngood work!

\n

but the paper claim that GNN performs better than MLP (which is used as a baseline)

", "rawMarkdown": "surprise that MLP can work.\ngood work!\n\nbut the paper claim that GNN performs better than MLP (which is used as a baseline)", "votes": null }, { "id": "2532285", "postDate": "11/20/2023 22:50:48", "content": "

very cool to pull this off with just an MLP. well done! Did you have to experiment much with the architecture? I wonder if you could share your thought process a bit more: how did you pick these feats? did you start with MLP as a \"start simple\" approach or was there another reason you dropped the GNNs?

", "rawMarkdown": "very cool to pull this off with just an MLP. well done! Did you have to experiment much with the architecture? I wonder if you could share your thought process a bit more: how did you pick these feats? did you start with MLP as a \"start simple\" approach or was there another reason you dropped the GNNs?", "votes": null }, { "id": "2532848", "postDate": "11/21/2023 11:35:25", "content": "

Simple and beautiful. Excellent.

\n

It is just so great to see you achieve great scores without \"fancy\" stuff - makes you think if it is really needed.

", "rawMarkdown": "Simple and beautiful. Excellent.\n\nIt is just so great to see you achieve great scores without \"fancy\" stuff - makes you think if it is really needed.", "votes": null }, { "id": "2533937", "postDate": "11/22/2023 09:55:47", "content": "

What paper you're talking about? Still, it's most likely uses node level MLP, while this solution uses some sort of edge level MLP. Considering that some top solutions reduce graph to \"1 node from configurable\", edge MLP is not so different from them.

", "rawMarkdown": "What paper you're talking about? Still, it's most likely uses node level MLP, while this solution uses some sort of edge level MLP. Considering that some top solutions reduce graph to \"1 node from configurable\", edge MLP is not so different from them.", "votes": null }, { "id": "2533948", "postDate": "11/22/2023 10:04:50", "content": "

Awesome solution.
\nI was wondering if you've tried using bi-directional edges? Like, edge_index.id.isin(data[\"node_config_ids\"] reduce connections to direct, but there are back connections edge_index.id_link.isin(data[\"node_config_ids\"] that can be useful too.

", "rawMarkdown": "Awesome solution.\nI was wondering if you've tried using bi-directional edges? Like, `edge_index.id.isin(data[\"node_config_ids\"]` reduce connections to direct, but there are back connections `edge_index.id_link.isin(data[\"node_config_ids\"]` that can be useful too.", "votes": null }, { "id": "2539726", "postDate": "11/27/2023 09:26:56", "content": "

How do you shrink node features to 12 values? In my solution, I have been able to squeeze data into 40 values based on data statistics.

", "rawMarkdown": "How do you shrink node features to 12 values? In my [solution](https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/457873), I have been able to squeeze data into 40 values based on data statistics.", "votes": null } ], "comments": [ { "id": 2531962, "author_name": "hengck23", "author_url": "", "post_date": "11/20/2023 17:12:06", "content": "

surprise that MLP can work.
\ngood work!

\n

but the paper claim that GNN performs better than MLP (which is used as a baseline)

", "votes": null, "replies": [ { "id": 2533937, "author_name": "yermvad", "author_url": "", "post_date": "11/22/2023 09:55:47", "content": "

What paper you're talking about? Still, it's most likely uses node level MLP, while this solution uses some sort of edge level MLP. Considering that some top solutions reduce graph to \"1 node from configurable\", edge MLP is not so different from them.

", "votes": null, "replies": [] } ] }, { "id": 2532285, "author_name": "thomasbfr", "author_url": "", "post_date": "11/20/2023 22:50:48", "content": "

very cool to pull this off with just an MLP. well done! Did you have to experiment much with the architecture? I wonder if you could share your thought process a bit more: how did you pick these feats? did you start with MLP as a \"start simple\" approach or was there another reason you dropped the GNNs?

", "votes": null, "replies": [] }, { "id": 2532848, "author_name": "narsil", "author_url": "", "post_date": "11/21/2023 11:35:25", "content": "

Simple and beautiful. Excellent.

\n

It is just so great to see you achieve great scores without \"fancy\" stuff - makes you think if it is really needed.

", "votes": null, "replies": [] }, { "id": 2533948, "author_name": "yermvad", "author_url": "", "post_date": "11/22/2023 10:04:50", "content": "

Awesome solution.
\nI was wondering if you've tried using bi-directional edges? Like, edge_index.id.isin(data[\"node_config_ids\"] reduce connections to direct, but there are back connections edge_index.id_link.isin(data[\"node_config_ids\"] that can be useful too.

", "votes": null, "replies": [] }, { "id": 2539726, "author_name": "belgraviton", "author_url": "", "post_date": "11/27/2023 09:26:56", "content": "

How do you shrink node features to 12 values? In my solution, I have been able to squeeze data into 40 values based on data statistics.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2531381": "`class SimpleMLP(torch.nn.Module):\n\n def __init__(self):\n super().__init__()\n self.embedding = torch.nn.Embedding(120,10)\n self.config_dense = torch.nn.Sequential(nn.Linear(128+8, 1024),\n nn.ReLU(),\n nn.Linear(1024, 128),\n nn.ReLU(),\n )\n self.node_dense = torch.nn.Sequential(nn.Linear(12*2+20, 1024),\n nn.ReLU(),\n nn.Linear(1024, 128),\n nn.ReLU(),\n )\n self.output=torch.nn.Sequential(nn.Linear(128*128+128, 1024),\n nn.ReLU(),\n nn.Linear(1024, 1),\n )\n def forward(self,x_cfg,x_feat,x_op):\n x_feat=x_feat.squeeze()\n x_op=x_op.squeeze()\n x_op = self.embedding(x_op).reshape(-1,20)\n x_feat = torch.concat([x_feat,x_op],dim =1)\n x_feat = self.node_dense(x_feat)\n x_graph = x_feat.unsqueeze(0)\n x_graph = x_graph.repeat_interleave(len(x_cfg),dim=0)\n x_cfg = torch.concat([x_cfg,x_graph],axis=2)\n x_cfg = self.config_dense(x_cfg)\n x=(x_feat.T@x_cfg).reshape(len(x_cfg),-1)\n x_cfg_mean=x_cfg.mean(dim=1)\n x = torch.concat([x,x_cfg_mean],axis=1)\n x=self.output(x)\n x=torch.flatten(x)\n return x\n \n\ndef load_data(row):\n\n node_feat_index=[21,22,23,24,28,\n 101,102,103,104,\n 134,135,136,\n ]\n config_index=[0,1,2,6,7,8,12,13]\n data= dict(np.load(row.path))\n X=data[\"node_feat\"][:,node_feat_index]\n node_feat=pd.DataFrame(X,columns=[\"fe%s\"% i for i in range(X.shape[1])])\n node_feat[\"id\"]=range(len(node_feat))\n node_feat_link=node_feat.copy()\n node_feat_link.columns=[i+\"_link\" for i in node_feat_link.columns]\n node_config_ids=pd.DataFrame({\"id\":data[\"node_config_ids\"],\"ind\":range(len(data[\"node_config_ids\"]))})\n node_opcode=pd.DataFrame({\"id\":range(len(data[\"node_opcode\"])),\"node_opcode\":data[\"node_opcode\"]})\n edge_index=pd.DataFrame(data[\"edge_index\"],columns=[\"id\",\"id_link\"])\n edge_index=edge_index[edge_index.id.isin(data[\"node_config_ids\"])]\n edge_index=edge_index.merge(node_opcode,on=\"id\",how=\"left\").merge(node_opcode.rename(columns={\"id\":\"id_link\",\"node_opcode\":\"node_opcode_link\"}),on=\"id_link\",how=\"left\")\n edge_index=edge_index.merge(node_config_ids,on=\"id\",how=\"left\")\n edge_index=edge_index.merge(node_feat,on=\"id\",how=\"left\").merge(node_feat_link,on=\"id_link\",how=\"left\")\n all_features=[i for i in edge_index.columns if i not in ['id', 'id_link', 'node_opcode','node_opcode_link', 'ind']]\n node_feat_array=edge_index[all_features].values.astype(np.float32)\n node_config_feat=data[\"node_config_feat\"][:,:,config_index]\n node_opcode_array=edge_index[[\"node_opcode\",\"node_opcode_link\"]].values\n node_config_feat=node_config_feat[:,edge_index[\"ind\"].values,:]\n label=data[\"config_runtime\"].argsort().argsort()/len(data[\"config_runtime\"])\n return {\"node_feat\":node_feat_array, #x_feat\n \"node_config_feat\":node_config_feat, #x_cfg\n \"node_opcode\":node_opcode_array, #x_op\n \"target\":label,\n \"config_runtime\":data[\"config_runtime\"],\n \"ind\":np.array(range(len(label)))\n }`", "2531962": "surprise that MLP can work.\ngood work!\n\nbut the paper claim that GNN performs better than MLP (which is used as a baseline)", "2532285": "very cool to pull this off with just an MLP. well done! Did you have to experiment much with the architecture? I wonder if you could share your thought process a bit more: how did you pick these feats? did you start with MLP as a \"start simple\" approach or was there another reason you dropped the GNNs?", "2532848": "Simple and beautiful. Excellent.\n\nIt is just so great to see you achieve great scores without \"fancy\" stuff - makes you think if it is really needed.", "2533937": "What paper you're talking about? Still, it's most likely uses node level MLP, while this solution uses some sort of edge level MLP. Considering that some top solutions reduce graph to \"1 node from configurable\", edge MLP is not so different from them.", "2533948": "Awesome solution.\nI was wondering if you've tried using bi-directional edges? Like, `edge_index.id.isin(data[\"node_config_ids\"]` reduce connections to direct, but there are back connections `edge_index.id_link.isin(data[\"node_config_ids\"]` that can be useful too.", "2539726": "How do you shrink node features to 12 values? In my [solution](https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/457873), I have been able to squeeze data into 40 values based on data statistics." }, "source": "meta" }