Did anyone manage to build a model to learn the layout data (I tried training on only one small file from Layout xla:default) without the TF script?
\nThings worked out really well in the tile:xla dataset but for some reason my model doesn't seem to learn anything in layout data (Loss not decreasing, cosine similarity between embedding vectors is 1).
\nIs it a gradient problem or something else? Did anyone else experience this problem?
I would love to get some insight on this.
\nThanks a lot.
Did anyone manage to build a model to learn the layout data (I tried training on only one small file from Layout xla:default) without the TF script?
\nThings worked out really well in the tile:xla dataset but for some reason my model doesn't seem to learn anything in layout data (Loss not decreasing, cosine similarity between embedding vectors is 1).
\nIs it a gradient problem or something else? Did anyone else experience this problem?
I would love to get some insight on this.
\nThanks a lot.
I am on the same boat, doesn't even overfit on a batch consiting of some of the models with less that 2000 nodes and only taking in to consideration the first 100 configs
", "rawMarkdown": "I am on the same boat, doesn't even overfit on a batch consiting of some of the models with less that 2000 nodes and only taking in to consideration the first 100 configs", "votes": null }, { "id": "2436939", "postDate": "09/13/2023 21:54:53", "content": "Yeah, the exact same thing happened to me. I tried to train it on only 100 configs but it doesn't seem to learn anything. I also tried many different models and loss functions to solve this. I thought I had a bug in my code.
", "rawMarkdown": "Yeah, the exact same thing happened to me. I tried to train it on only 100 configs but it doesn't seem to learn anything. I also tried many different models and loss functions to solve this. I thought I had a bug in my code.", "votes": null }, { "id": "2438161", "postDate": "09/14/2023 05:04:26", "content": "In their dataset paper, the competition hosts described the different GNNs used for layout and tile problems. I don't think it's viable to use the tile model for the layout problem since the data structure is quite different.
", "rawMarkdown": "In their dataset paper, the competition hosts described the different GNNs used for layout and tile problems. I don't think it's viable to use the tile model for the layout problem since the data structure is quite different.", "votes": null }, { "id": "2439118", "postDate": "09/14/2023 16:40:57", "content": "You're right. But I tried the things presented in this paper (without GST, but I tried on a file with 372 nodes. In the paper they say that they use a segment size of 1000 so I suppose it shouldn't matter much) and I didn't get any learning (not even on 100 examples out of the file).
", "rawMarkdown": "You're right. But I tried the things presented in this paper (without GST, but I tried on a file with 372 nodes. In the paper they say that they use a segment size of 1000 so I suppose it shouldn't matter much) and I didn't get any learning (not even on 100 examples out of the file).", "votes": null }, { "id": "2443471", "postDate": "09/17/2023 18:15:13", "content": "In our experiments, we found a couple important things:
\nI hope this helps!
", "rawMarkdown": "In our experiments, we found a couple important things:\n- Using pairwise/listwise rank loss (instead of a typical MSE/MAPE loss) is important.\n- Combine the layout configuration with node features before feeding into GNN is important.\n- When concatenating layout config with node features, the model learns better when we scale up the layout portion, e.g. concat(node_features, SCALE * layout_config), where SCALE is something like 100.\n\nI hope this helps!", "votes": null }, { "id": "2444485", "postDate": "09/18/2023 09:43:21", "content": "Hello, thanks for your tips, but I have several questions. When I combine the layout configuration with node features, it can't work, because c is too big, and I need combine c times. May I ask if you have any solution.
", "rawMarkdown": "Hello, thanks for your tips, but I have several questions. When I combine the layout configuration with node features, it can't work, because c is too big, and I need combine c times. May I ask if you have any solution.", "votes": null }, { "id": "2447388", "postDate": "09/20/2023 03:48:14", "content": "c is the number of difference configurations for a graph. When training a model, we don't need to train all c samples in a single training batch.
Thank you very much! It helps me a lot.
", "rawMarkdown": "Thank you very much! It helps me a lot.", "votes": null }, { "id": "2451273", "postDate": "09/22/2023 12:33:48", "content": "Thanks a lot! I have a few more questions:
\nIn the GST paper, it is written that for the dataset TPUGraphs \"the prediction head is part of F, and F' is simply a summation function.\" So how is the \"Stale Embedding Dropout\" implemented if the output of the model is a size 1 vector? Do we zero the entire output of the segment? Or is the output of the model an embedding vector (the TPUGraphs dataset paper says it outputs an embedding vector so it's quite puzzling).
Also in the GST paper it is not mentioned how many segments are trained for each batch. For N different segments, approximately how many segments will require gradient, and how many will be fetched from the historical embedding table?
What is the process training? In the GST paper it says that for each epoch only one batch is calculated. Is that batch taken from only one file that is sampled randomly? All the possible pairs in the batch are computed to produce a loss (for example N=8 we compute 8*7/2 = 28 pairs)?
F outputs an embedding vector.
The number of segments are variable, depending on the graph size. We partition a graph such that a segment has < 1000 nodes.
In each training step, one batch of data is used (one graph and multiple configs, selected randomly). Within each batch, we compute either pairwise hinge loss, pairwise logistic loss, or list MLE.
I am on the same boat, doesn't even overfit on a batch consiting of some of the models with less that 2000 nodes and only taking in to consideration the first 100 configs
", "votes": null, "replies": [ { "id": 2436939, "author_name": "amitaharoni", "author_url": "", "post_date": "09/13/2023 21:54:53", "content": "Yeah, the exact same thing happened to me. I tried to train it on only 100 configs but it doesn't seem to learn anything. I also tried many different models and loss functions to solve this. I thought I had a bug in my code.
", "votes": null, "replies": [] } ] }, { "id": 2438161, "author_name": "passengerc07", "author_url": "", "post_date": "09/14/2023 05:04:26", "content": "In their dataset paper, the competition hosts described the different GNNs used for layout and tile problems. I don't think it's viable to use the tile model for the layout problem since the data structure is quite different.
", "votes": null, "replies": [ { "id": 2439118, "author_name": "amitaharoni", "author_url": "", "post_date": "09/14/2023 16:40:57", "content": "You're right. But I tried the things presented in this paper (without GST, but I tried on a file with 372 nodes. In the paper they say that they use a segment size of 1000 so I suppose it shouldn't matter much) and I didn't get any learning (not even on 100 examples out of the file).
", "votes": null, "replies": [] } ] }, { "id": 2443471, "author_name": "mangpophothilimthana", "author_url": "", "post_date": "09/17/2023 18:15:13", "content": "In our experiments, we found a couple important things:
\nI hope this helps!
", "votes": null, "replies": [ { "id": 2444485, "author_name": "ljjsxx", "author_url": "", "post_date": "09/18/2023 09:43:21", "content": "Hello, thanks for your tips, but I have several questions. When I combine the layout configuration with node features, it can't work, because c is too big, and I need combine c times. May I ask if you have any solution.
", "votes": null, "replies": [ { "id": 2447388, "author_name": "mangpophothilimthana", "author_url": "", "post_date": "09/20/2023 03:48:14", "content": "c is the number of difference configurations for a graph. When training a model, we don't need to train all c samples in a single training batch.
Thank you very much! It helps me a lot.
", "votes": null, "replies": [] } ] } ] }, { "id": 2451273, "author_name": "amitaharoni", "author_url": "", "post_date": "09/22/2023 12:33:48", "content": "Thanks a lot! I have a few more questions:
\nIn the GST paper, it is written that for the dataset TPUGraphs \"the prediction head is part of F, and F' is simply a summation function.\" So how is the \"Stale Embedding Dropout\" implemented if the output of the model is a size 1 vector? Do we zero the entire output of the segment? Or is the output of the model an embedding vector (the TPUGraphs dataset paper says it outputs an embedding vector so it's quite puzzling).
Also in the GST paper it is not mentioned how many segments are trained for each batch. For N different segments, approximately how many segments will require gradient, and how many will be fetched from the historical embedding table?
What is the process training? In the GST paper it says that for each epoch only one batch is calculated. Is that batch taken from only one file that is sampled randomly? All the possible pairs in the batch are computed to produce a loss (for example N=8 we compute 8*7/2 = 28 pairs)?
F outputs an embedding vector.
The number of segments are variable, depending on the graph size. We partition a graph such that a segment has < 1000 nodes.
In each training step, one batch of data is used (one graph and multiple configs, selected randomly). Within each batch, we compute either pairwise hinge loss, pairwise logistic loss, or list MLE.