\n
![\"Screenshot-2023-10-19-at-13-06-48-Weights-Biases.png\"](\"https://i.postimg.cc/htV15yP5/Screenshot-2023-10-19-at-13-06-48-Weights-Biases.png\")
Hi everyone,
\nCurrently, I run simple experiments on layout collection using Graph Segment Training (GST). Some weird training curve behaviors confuse me a lot. As shown in the figure below. The training loss keeps increasing with validation performance improving (though the curve is shaky…).
\n
\n\nRed curve for nlp-random, purple for nlp-default, green for xla-random, brown for xla-default
\n
Experimental setup is described as follows:
\nAdam(lr=1e-3, weight_decay=0) with grad norm clipped at 1e-2train / valid splittingDoes anyone encounter the same issue while applying GST? My first guess is that the staleness issue of graph segment embedding is a drag on training loss. Or, maybe I did something wrong.
\nPlease feel free to comment your idea here, thanks a lot!
", "messages": [ { "id": 2497447, "postDate": "2023-10-24T16:11:51.687Z", "content": "I also observed similar behavior during my past experiments, training loss will decrease for a while and keep increasing afterward. After reading your post I started looking further into this problem and found out that there may be some error in the official GST implementation regarding the dropout scaling. In the official GST implementation, the author always assumes half of the graph segments sampled from cache will be zeros, but in reality, that will only be true after 2000+ epochs(if we train with batch size 8 & 32 configs per graph on xla-default subset), most of the cached graph segment embedding will be 0 before that. So most of the time during training our model uses graph embedding that is \"wrong\" in various degrees.
\nAfter I adjust the way GST dropout scaling is computed, the problem seems to be gone.
\nHere is the result:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2295998%2F21aa487a78a84e828871cd125a8dcaed%2FScreenshot%20from%202023-10-24%2023-51-22.png?generation=1698162801754216&alt=media\")
And unfortunately, fixing the weird training loss dynamic doesn't come with the benefit of a higher validation opa score.🤣
\nBut I guess that is the property of this dataset, you can make a decent prediction with only part of graph, so using the wrong graph segment embedding from cache also doesn't hurt the accuracy too much.
Very helpful insight, thank you so much for sharing this.
", "rawMarkdown": "Very helpful insight, thank you so much for sharing this.", "votes": 1 }, { "id": 2497732, "postDate": "2023-10-24T20:45:01.370Z", "content": "Hi @dsfhe49854,
\nThanks very much for your sharing. I didn't notice this issue. Because the embedding table is initialized with all-zeros, the full-graph context obtained from stale embedding dropout actually incorporates too much non-updated information during early iterations. I'll try to fix it and see if it's effective for me.
", "rawMarkdown": "Hi @dsfhe49854,\n\nThanks very much for your sharing. I didn't notice this issue. Because the embedding table is initialized with all-zeros, the full-graph context obtained from stale embedding dropout actually incorporates too much non-updated information during early iterations. I'll try to fix it and see if it's effective for me.", "votes": 1, "replies": [ { "id": 2504809, "postDate": "2023-10-30T06:56:53.440Z", "content": "Hi @abaojiang ,
\nDid you make any changes regarding this issue? I also noticed that there are a lot of zeros in the graph embeddings initially, and they decrease over time. I'm not really sure how this relates to the dropout strategy. Looking forward to hearing from you.
Hi @chitra077,
\nI haven't had time to fix this issue. But, what I'll do next is summarized as follows,
\nHope this helps.
", "rawMarkdown": "Hi @chitra077,\n\nI haven't had time to fix this issue. But, what I'll do next is summarized as follows,\n1. Disable stale embedding table to see if it really takes effect.\n * This means the full-graph context is always equal to the only segment embedding.\n2. If stale embedding table really works, then I would like to try to update embedding table periodically.\n * That is, every entry is updated by a simple forward pass every X iterations. However, this will increase the time cost.\n\nHope this helps." } ] }, { "id": 2501035, "postDate": "2023-10-27T07:39:52.483Z", "content": "Hi. I appreciate your informative information. Could you share your insight for adjusting dropout scaling implementation?
", "rawMarkdown": "Hi. I appreciate your informative information. Could you share your insight for adjusting dropout scaling implementation?" }, { "id": 2504501, "postDate": "2023-10-29T23:47:39.933Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2508354, "postDate": "2023-11-01T16:11:21.617Z", "content": "@sunjongpark My implementation look something like this:
\n (batch_other) > :\n batch_other = torch.cat(batch_other, dim=)\n mask = binomial.sample((batch_other.shape[], )).to(torch.device(cfg.device))\n batch_other = batch_other.to(torch.device(cfg.device))\n batch_other = batch_other * mask \n batch_other_embed = torch.zeros_like(graph_embed)\n batch_zero_embed = torch.zeros_like(graph_embed)\n part_cnt = \n i, num_parts (batch_num_parts):\n j (num_parts-):\n batch_other_embed[i, :] += batch_other[part_cnt, :]\n batch_zero_embed[i] += torch.(batch_other[part_cnt, :]).() < \n part_cnt += \n batch_num_parts = torch.Tensor(batch_num_parts).to(torch.device(cfg.device))\n batch_num_parts = batch_num_parts.view(-, )\n multiplier_num = batch_num_parts / (batch_num_parts - batch_zero_embed)\n pred = graph_embed * multiplier_num + batch_other_embed\nYou should able to replace this block of code from the original implementation and check the effect yourself.
", "rawMarkdown": "@sunjongpark My implementation look something like this:\n```python\nif len(batch_other) > 0:\n batch_other = torch.cat(batch_other, dim=0)\n mask = binomial.sample((batch_other.shape[0], 1)).to(torch.device(cfg.device))\n batch_other = batch_other.to(torch.device(cfg.device))\n batch_other = batch_other * mask \n batch_other_embed = torch.zeros_like(graph_embed)\n batch_zero_embed = torch.zeros_like(graph_embed)\n part_cnt = 0\n for i, num_parts in enumerate(batch_num_parts):\n for j in range(num_parts-1):\n batch_other_embed[i, :] += batch_other[part_cnt, :]\n batch_zero_embed[i] += torch.abs(batch_other[part_cnt, :]).sum() < 1e-6\n part_cnt += 1\n batch_num_parts = torch.Tensor(batch_num_parts).to(torch.device(cfg.device))\n batch_num_parts = batch_num_parts.view(-1, 1)\n multiplier_num = batch_num_parts / (batch_num_parts - batch_zero_embed)\n pred = graph_embed * multiplier_num + batch_other_embed\n```\nYou should able to replace [this block of code](https://github.com/kaidic/GST/blob/7183b314cc8567e95702d62d726bbe270fa36fd6/graphgps/train/custom_tpu_train.py#L128:L142) from the original implementation and check the effect yourself.", "votes": 2, "replies": [ { "id": 2514031, "postDate": "2023-11-06T01:09:50.653Z", "content": "THANKS! VERY NICE IDEA!
", "rawMarkdown": "THANKS! VERY NICE IDEA!" } ] } ] }, { "id": 2488621, "postDate": "2023-10-19T11:27:55.600Z", "content": "Hi everyone,
\nCurrently, I run simple experiments on layout collection using Graph Segment Training (GST). Some weird training curve behaviors confuse me a lot. As shown in the figure below. The training loss keeps increasing with validation performance improving (though the curve is shaky…).
\n
\n\nRed curve for nlp-random, purple for nlp-default, green for xla-random, brown for xla-default
\n
Experimental setup is described as follows:
\nAdam(lr=1e-3, weight_decay=0) with grad norm clipped at 1e-2train / valid splittingDoes anyone encounter the same issue while applying GST? My first guess is that the staleness issue of graph segment embedding is a drag on training loss. Or, maybe I did something wrong.
\nPlease feel free to comment your idea here, thanks a lot!
", "rawMarkdown": "Hi everyone,\n\nCurrently, I run simple experiments on *layout* collection using **Graph Segment Training (GST)**. Some weird training curve behaviors confuse me a lot. As shown in the figure below. The training loss keeps increasing with validation performance improving (though the curve is shaky...).\n[](https://postimg.cc/SnN9XG96)\n> Red curve for *nlp-random*, purple for *nlp-default*, green for *xla-random*, brown for *xla-default*\n\nExperimental setup is described as follows:\n1. Batch size: 8 with gradient accumulation step 4 (*i.e.,* equal to bs 32)\n2. Loss criterion: Pairwise hinge loss\n3. Optimizer: `Adam(lr=1e-3, weight_decay=0)` with grad norm clipped at `1e-2`\n4. Learning scheduler: Cosine decay w/o warmup\n5. Training strategy: GST w/ each segment containing < 1000 nodes\n * Combine with **Historical Embedding Table** and **Stale Embedding Dropout**\n6. CV scheme: Official `train` / `valid` splitting\n\nDoes anyone encounter the same issue while applying **GST**? My first guess is that the **staleness** issue of graph segment embedding is a drag on training loss. Or, maybe I did something wrong.\n\nPlease feel free to comment your idea here, thanks a lot!", "votes": 7 }, { "id": 2491698, "postDate": "2023-10-21T22:58:29.907Z", "content": "The truth is that after applying GST, the result gets worse. Train loss and val loss fluctuate in a small zone, it's really confused.
", "rawMarkdown": "The truth is that after applying GST, the result gets worse. Train loss and val loss fluctuate in a small zone, it's really confused.", "votes": 3, "replies": [ { "id": 2492222, "postDate": "2023-10-22T10:35:13.763Z", "content": "Hi @lizhecheng,
\nConsidering the limitation on GPU memory, I tried GST directly for layout. The learning curves generally look like what I posted above. After playing around with some hyperparameters, like stale embedding dropout, both training and validation losses decrease for nlp-random only. As for other metrics, all 4 layout collections have gradually improving OPAs. However, I’m still confused about the behavior of the training loss now. I’ll share more charts later.
", "rawMarkdown": "Hi @lizhecheng,\n\nConsidering the limitation on GPU memory, I tried GST directly for layout. The learning curves generally look like what I posted above. After playing around with some hyperparameters, like stale embedding dropout, both training and validation losses decrease for *nlp-random* only. As for other metrics, all 4 layout collections have gradually improving OPAs. However, I’m still confused about the behavior of the training loss now. I’ll share more charts later.", "votes": 1, "replies": [ { "id": 2492497, "postDate": "2023-10-22T15:18:45.127Z", "content": "Thank you very much, we will continue exploring it.
", "rawMarkdown": "Thank you very much, we will continue exploring it.", "votes": 1 }, { "id": 2492767, "postDate": "2023-10-22T21:29:36.570Z", "content": "Hi @lizhecheng,
\nI just tried another experiment to see the influence of modeling randomness (i.e., train models with another random seed with exactly the same validation setup). I chose layout-xla-default as an example. As can be seen, just changing the random seed led to a significantly different training process. Both training and validation losses decrease, though the training part seems within a narrower range. Also, kendall tau jumps about 1.5x. Hence, more robust CV scheme should be established. And, the training loss behavior still remains a mystery for me…
\n![\"Screenshot-2023-10-22-at-22-12-51-Weights-Biases.png\"](\"https://i.postimg.cc/Z5DgdHmf/Screenshot-2023-10-22-at-22-12-51-Weights-Biases.png\")
So it means the only thing that we can do now is to change the random seed and expect a better result 😭
", "rawMarkdown": "So it means the only thing that we can do now is to change the random seed and expect a better result 😭", "votes": 1 }, { "id": 2492917, "postDate": "2023-10-23T03:08:26.380Z", "content": "is it correct to assume that the amount of training data increased substantially as well? the loss value was two digits previously. or did you change your loss function as well?
", "rawMarkdown": "is it correct to assume that the amount of training data increased substantially as well? the loss value was two digits previously. or did you change your loss function as well?", "votes": 2 }, { "id": 2492924, "postDate": "2023-10-23T03:19:06.197Z", "content": "Can you be more specific? I don't really understand your comment.
", "rawMarkdown": "Can you be more specific? I don't really understand your comment.", "votes": 2 }, { "id": 2493824, "postDate": "2023-10-23T15:22:03.900Z", "content": "Hi @lizhecheng,
\nI think the intrinsically small dataset is the problem leading to instability. The intention of sharing the charts is that I want to show the randomness matters a lot. I still try to find ways to mitigate the performance fluctuation (i.e., narrow the variance w.r.t. random seeds). However, I haven't had an answer now.
", "rawMarkdown": "Hi @lizhecheng,\n\nI think the intrinsically small dataset is the problem leading to instability. The intention of sharing the charts is that I want to show the randomness matters a lot. I still try to find ways to mitigate the performance fluctuation (*i.e.,* narrow the variance w.r.t. random seeds). However, I haven't had an answer now.", "votes": 1 }, { "id": 2493830, "postDate": "2023-10-23T15:26:05.503Z", "content": "Thank you so much. Will you good luck, we are also doing this now.
", "rawMarkdown": "Thank you so much. Will you good luck, we are also doing this now.", "votes": 1 }, { "id": 2493852, "postDate": "2023-10-23T15:50:02.760Z", "content": "Hi @rob1080ti,
\nSorry for not clearly explaining the difference. The previous experiments with lower absolute loss values are trained with 32 sampled configurations per segment. In new experiments, I enlarge to 200 sampled configurations per segment. I don't change the loss function here.
\nAt least for me, the fluctuation of performance at different checkpoints is somewhat mitigated but not much. Hope this helps!
thank you! i have also tried more configurations which has helped but is expensive if trying to rapidly test other strategies in combination. one idea i had which may help the randomness issue is taking multiple checkpoints as topk checkpoints and running inference with these and try selecting based on the modeled overall time for all 1000 configs per file. if this was meaningful, it could also enable per file selections instead of per model checkpoint selections. how the config runtimes are scaled may or may not be important and regardless, this experiment may be very expensive and hence the topk checkpoints idea. also, it has occurred to me that dramatic changes in terms of configuration parameters may change absolute expected runtime costs so this may be very brittle if used across runs. (note: i have not tried this and i do not expect much time for further experimentation)
", "rawMarkdown": "thank you! i have also tried more configurations which has helped but is expensive if trying to rapidly test other strategies in combination. one idea i had which may help the randomness issue is taking multiple checkpoints as topk checkpoints and running inference with these and try selecting based on the modeled overall time for all 1000 configs per file. if this was meaningful, it could also enable per file selections instead of per model checkpoint selections. how the config runtimes are scaled may or may not be important and regardless, this experiment may be very expensive and hence the topk checkpoints idea. also, it has occurred to me that dramatic changes in terms of configuration parameters may change absolute expected runtime costs so this may be very brittle if used across runs. (note: i have not tried this and i do not expect much time for further experimentation)", "votes": 1 }, { "id": 2493884, "postDate": "2023-10-23T16:11:59.133Z", "content": "Hi @rdxsun,
\nSorry for confusing you. In the original post, I just ask if someone faces the similar issue as me (i.e., increasing training loss with decreasing val loss). The experimental settings are summarized above. What I forgot to mention was the number of sampled configurations used in Graph Segment Training, which was set to 32 configs for each segment with stale embedding dropout 0.5 in the original experiments.
\nAfter increasing the number of configs per segment to 200 in new experiments, I find out it helps stabilize the training process in terms of OPA (i.e., performance becomes less shaky). The potential reason behind the scene is that the small number of sampled configurations makes the training process too random to converge. Also, I believe better config sampling strategies can help (e.g., always considering the extreme runtime cases), which I'll try next.
\nAs for this thread, I want to share the important role the randomness plays here. Just changing the random seed can lead to significantly different performance. Hence, a more robust validation scheme should be established. Or, it's hard to trust what the validation set tells me, which might be caused by a merely lucky / unlucky seed.
", "rawMarkdown": "Hi @rdxsun,\n\nSorry for confusing you. In the original post, I just ask if someone faces the similar issue as me (*i.e.,* increasing training loss with decreasing val loss). The experimental settings are summarized above. What I forgot to mention was the **number of sampled configurations** used in **Graph Segment Training**, which was set to 32 configs for each segment with stale embedding dropout 0.5 in the original experiments.\n\nAfter increasing the number of configs per segment to 200 in new experiments, I find out it helps stabilize the training process in terms of OPA (*i.e.,* performance becomes less shaky). The potential reason behind the scene is that the small number of sampled configurations makes the training process too random to converge. Also, I believe better config sampling strategies can help (*e.g.,* always considering the extreme runtime cases), which I'll try next.\n\nAs for this thread, I want to share the important role the randomness plays here. Just changing the random seed can lead to significantly different performance. Hence, a more robust validation scheme should be established. Or, it's hard to trust what the validation set tells me, which might be caused by a merely lucky / unlucky seed.", "votes": 1 }, { "id": 2493892, "postDate": "2023-10-23T16:16:21.517Z", "content": "Sorry, my teammate wanted to reply to Rob, it was a mistake, we can understand what you shared. Thank you so much.🤣💪
", "rawMarkdown": "Sorry, my teammate wanted to reply to Rob, it was a mistake, we can understand what you shared. Thank you so much.🤣💪", "votes": 1 }, { "id": 2493904, "postDate": "2023-10-23T16:24:56.800Z", "content": "Oh it's okay 🤣, I was just afraid that I didn't explain well. Good luck to you!!
", "rawMarkdown": "Oh it's okay 🤣, I was just afraid that I didn't explain well. Good luck to you!!", "votes": 1 }, { "id": 2493925, "postDate": "2023-10-23T16:32:30.910Z", "content": "Your topic is really helpful and valuable, thanks so much. ❤️
", "rawMarkdown": "Your topic is really helpful and valuable, thanks so much. ❤️", "votes": 2 }, { "id": 2493972, "postDate": "2023-10-23T16:48:31.923Z", "content": "Sorry, I wasn't sure if their question was directed at me. 😅
", "rawMarkdown": "Sorry, I wasn't sure if their question was directed at me. 😅", "votes": 1 }, { "id": 2494004, "postDate": "2023-10-23T17:01:00.213Z", "content": "Thanks for your great comments, now we have understood this is it correct to assume that the amount of training data increased substantially as well? the loss value was two digits previously. or did you change your loss function as well?
I think the opa/kt challenges are experienced across the board (see https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/443415 ). It mirrors my experience too, and I'm using a modified starter instead of the GST EFD repo.
\nSome thoughts: top teams may be cleaning the data, adding the additional features from the pb but not in the npz, combining the gnn with another approach, downsampling the training set (unlikely to improve opa/kt directly but may speed up iterating over experiments)
", "rawMarkdown": "I think the opa/kt challenges are experienced across the board (see https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/443415 ). It mirrors my experience too, and I'm using a modified starter instead of the GST EFD repo.\n\nSome thoughts: top teams may be cleaning the data, adding the additional features from the pb but not in the npz, combining the gnn with another approach, downsampling the training set (unlikely to improve opa/kt directly but may speed up iterating over experiments)", "votes": 1, "replies": [ { "id": 2489919, "postDate": "2023-10-20T09:45:29.993Z", "content": "Hi @rob1080ti,
\nYeah, I also read that post and it seems that we're facing the similar challenges. Because I also had my own implementation of GST+EFD, I couldn't ensure that it's bug-free or not 😅. However, I drew some insights from another quick experiment. When I decrease dropout ratio in Stale Embedding Dropout from 0.5 to 0.25, the new learning curves are as follows,
\n![\"Screenshot-2023-10-20-at-11-41-49-Weights-Biases.png\"](\"https://i.postimg.cc/wv50TGHj/Screenshot-2023-10-20-at-11-41-49-Weights-Biases.png\")
I'll share more observations if I find something interesting (and if time allows…). Also, thanks for sharing your thoughts.
", "rawMarkdown": "Hi @rob1080ti,\n\nYeah, I also read that post and it seems that we're facing the similar challenges. Because I also had my own implementation of GST+EFD, I couldn't ensure that it's bug-free or not 😅. However, I drew some insights from another quick experiment. When I decrease dropout ratio in **Stale Embedding Dropout** from 0.5 to 0.25, the new learning curves are as follows,\n[](https://postimg.cc/Kkzn0QR6)\n\nI'll share more observations if I find something interesting (and if time allows...). Also, thanks for sharing your thoughts.", "votes": 1, "replies": [ { "id": 2490040, "postDate": "2023-10-20T11:49:04.947Z", "content": "i haven't really found a lot of benefit in tinkering with dropout. i'm also surprised by your training loss increase; i'm also using pairwise hinge for the layout models and training and validation losses both decrease for me. however, you may want to look at whether you are handling batching properly with the metrics (if you are using batching). i initially had some wacky results with kt before i realized that.
", "rawMarkdown": "i haven't really found a lot of benefit in tinkering with dropout. i'm also surprised by your training loss increase; i'm also using pairwise hinge for the layout models and training and validation losses both decrease for me. however, you may want to look at whether you are handling batching properly with the metrics (if you are using batching). i initially had some wacky results with kt before i realized that.", "votes": 1 }, { "id": 2490553, "postDate": "2023-10-20T19:23:02.610Z", "content": "Hi @rob1080ti,
\nI'll check if there's any bug in my code, thanks a lot!
", "rawMarkdown": "Hi @rob1080ti,\n\nI'll check if there's any bug in my code, thanks a lot!" } ] } ] }, { "id": 2489724, "postDate": "2023-10-20T07:04:39.263Z", "content": "How does your train performance look like?
", "rawMarkdown": "How does your train performance look like? ", "votes": 2, "replies": [ { "id": 2489923, "postDate": "2023-10-20T09:47:19.143Z", "content": "Hi @amitaharoni,
\nRunning evaluation on training set with different metrics is on my list and I'll share when experiments are done! Did you observe the similar behavior on the learning curve?
", "rawMarkdown": "Hi @amitaharoni,\n\nRunning evaluation on training set with different metrics is on my list and I'll share when experiments are done! Did you observe the similar behavior on the learning curve?", "votes": 2, "replies": [ { "id": 2489931, "postDate": "2023-10-20T09:58:20.603Z", "content": "To be honest I tried to run GST (all forms presented in the paper) and it just worsen my results. But I believe my implementation wasn't good. I might get back to it again later
", "rawMarkdown": "To be honest I tried to run GST (all forms presented in the paper) and it just worsen my results. But I believe my implementation wasn't good. I might get back to it again later", "votes": 2 }, { "id": 2490030, "postDate": "2023-10-20T11:41:24.477Z", "content": "btw for kt and opa i've noticed that approximations of each other look like kt=(opa-0.5)*2 and opa=(kt+1)/2
", "rawMarkdown": "btw for kt and opa i've noticed that approximations of each other look like kt=(opa-0.5)*2 and opa=(kt+1)/2", "votes": 2 }, { "id": 2490568, "postDate": "2023-10-20T19:42:16.873Z", "content": "It's exactly the relationship between these two metrics. Let c be the number of concordant pairs and d the discordant ones. These two metrics can be formulated as follows,
\n$$OPA = \\frac{c}{c + d}$$
\n$$Kendall\\ \\tau = \\frac{c - d}{c + d}$$
Hence, the following relationship between the two holds,
\n$$OPA = \\frac{Kendall\\ \\tau + 1}{2}$$
May I ask how is the result now? Any improvement after using GST?
", "rawMarkdown": "May I ask how is the result now? Any improvement after using GST?", "votes": 1 } ] } ] }, { "id": 2508380, "postDate": "2023-11-01T16:37:23.793Z", "content": "raise KeyError( \"Non-existent config key: f]\" . format(full key))
\nKeyError: ' Non-existent config key: train. mode
\nSorry i new to here.can i ask why my gst had this problem
", "rawMarkdown": "raise KeyError( \"Non-existent config key: f]\" . format(full key))\n\nKeyError: ' Non-existent config key: train. mode \n\nSorry i new to here.can i ask why my gst had this problem" }, { "id": 2492914, "postDate": "2023-10-23T03:07:35.193Z", "content": "Hello,
\nAfter reviewing the TPUGRAPHS paper and your work, I'm somewhat perplexed. The paper emphasizes that by applying GST, it can address the issue of memory insufficiency while also improving accuracy relative to operations on full graphs. However, the content demonstrated in your charts seems to suggest the opposite outcome. Are there any potential explanations for this, or alternative implementation methods you might suggest?
\nAdditionally, I would like to inquire about your training model. If GST is not employed, what model do you utilize for training purposes?
\nThank you！
", "rawMarkdown": "Hello,\n\nAfter reviewing the TPUGRAPHS paper and your work, I'm somewhat perplexed. The paper emphasizes that by applying GST, it can address the issue of memory insufficiency while also improving accuracy relative to operations on full graphs. However, the content demonstrated in your charts seems to suggest the opposite outcome. Are there any potential explanations for this, or alternative implementation methods you might suggest?\n\nAdditionally, I would like to inquire about your training model. If GST is not employed, what model do you utilize for training purposes?\n\nThank you！", "votes": 1, "isDeleted": true, "replies": [ { "id": 2496996, "postDate": "2023-10-24T11:42:16.073Z", "content": "Hi @mstarlioning,
\nAs I mentioned above, considering the GPU memory limitation, I didn't even manage to implement full-graph training. My charts contain learning curves for 4 different types of layout collections, all of which are trained with GST framework. Hence, it's hard for me to comment on the comparison between full-graph training GST. But, maybe someone with sufficient hardware spec can help answer!
", "rawMarkdown": "Hi @mstarlioning,\n\nAs I mentioned above, considering the GPU memory limitation, I didn't even manage to implement full-graph training. My charts contain learning curves for 4 different types of *layout* collections, all of which are trained with **GST** framework. Hence, it's hard for me to comment on the comparison between full-graph training **GST**. But, maybe someone with sufficient hardware spec can help answer!", "replies": [ { "id": 2497183, "postDate": "2023-10-24T13:56:56.220Z", "content": "@abaojiang -- have you noticed a speedup with GPU training? With tfgnn, it is slower and a lot faster to just use CPU (this seemed to be the case with a 1080ti + i7 7700k).
", "rawMarkdown": "@abaojiang -- have you noticed a speedup with GPU training? With tfgnn, it is slower and a lot faster to just use CPU (this seemed to be the case with a 1080ti + i7 7700k).", "votes": 1 }, { "id": 2497727, "postDate": "2023-10-24T20:25:58.007Z", "content": "Hi @rob1080ti,
\nBecause I need to balance the memory usage of CPU and GPU, I don't train models using CPU. Also, I don't use tfgnn framework, I rewrite the pipeline with pure torch. So, I think I can't comment on this topic.
I also observed similar behavior during my past experiments, training loss will decrease for a while and keep increasing afterward. After reading your post I started looking further into this problem and found out that there may be some error in the official GST implementation regarding the dropout scaling. In the official GST implementation, the author always assumes half of the graph segments sampled from cache will be zeros, but in reality, that will only be true after 2000+ epochs(if we train with batch size 8 & 32 configs per graph on xla-default subset), most of the cached graph segment embedding will be 0 before that. So most of the time during training our model uses graph embedding that is \"wrong\" in various degrees.
\nAfter I adjust the way GST dropout scaling is computed, the problem seems to be gone.
\nHere is the result:
\n
And unfortunately, fixing the weird training loss dynamic doesn't come with the benefit of a higher validation opa score.🤣
\nBut I guess that is the property of this dataset, you can make a decent prediction with only part of graph, so using the wrong graph segment embedding from cache also doesn't hurt the accuracy too much.
Very helpful insight, thank you so much for sharing this.
", "votes": 1, "replies": [] }, { "id": 2497732, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-10-24T20:45:01.370000", "content": "Hi @dsfhe49854,
\nThanks very much for your sharing. I didn't notice this issue. Because the embedding table is initialized with all-zeros, the full-graph context obtained from stale embedding dropout actually incorporates too much non-updated information during early iterations. I'll try to fix it and see if it's effective for me.
", "votes": 1, "replies": [ { "id": 2504809, "author_name": "Chitra077", "author_url": "", "post_date": "2023-10-30T06:56:53.440000", "content": "Hi @abaojiang ,
\nDid you make any changes regarding this issue? I also noticed that there are a lot of zeros in the graph embeddings initially, and they decrease over time. I'm not really sure how this relates to the dropout strategy. Looking forward to hearing from you.
Hi @chitra077,
\nI haven't had time to fix this issue. But, what I'll do next is summarized as follows,
\nHope this helps.
", "votes": 0, "replies": [] } ] }, { "id": 2501035, "author_name": "SunJong Park", "author_url": "", "post_date": "2023-10-27T07:39:52.483000", "content": "Hi. I appreciate your informative information. Could you share your insight for adjusting dropout scaling implementation?
", "votes": 0, "replies": [] }, { "id": 2504501, "author_name": "", "author_url": "", "post_date": "2023-10-29T23:47:39.933000", "content": "", "votes": 0, "replies": [] }, { "id": 2508354, "author_name": "RZ", "author_url": "", "post_date": "2023-11-01T16:11:21.617000", "content": "@sunjongpark My implementation look something like this:
\n (batch_other) > :\n batch_other = torch.cat(batch_other, dim=)\n mask = binomial.sample((batch_other.shape[], )).to(torch.device(cfg.device))\n batch_other = batch_other.to(torch.device(cfg.device))\n batch_other = batch_other * mask \n batch_other_embed = torch.zeros_like(graph_embed)\n batch_zero_embed = torch.zeros_like(graph_embed)\n part_cnt = \n i, num_parts (batch_num_parts):\n j (num_parts-):\n batch_other_embed[i, :] += batch_other[part_cnt, :]\n batch_zero_embed[i] += torch.(batch_other[part_cnt, :]).() < \n part_cnt += \n batch_num_parts = torch.Tensor(batch_num_parts).to(torch.device(cfg.device))\n batch_num_parts = batch_num_parts.view(-, )\n multiplier_num = batch_num_parts / (batch_num_parts - batch_zero_embed)\n pred = graph_embed * multiplier_num + batch_other_embed\nYou should able to replace this block of code from the original implementation and check the effect yourself.
", "votes": 2, "replies": [ { "id": 2514031, "author_name": "Jianhua Yu", "author_url": "", "post_date": "2023-11-06T01:09:50.653000", "content": "THANKS! VERY NICE IDEA!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2491698, "author_name": "Zhecheng Li", "author_url": "", "post_date": "2023-10-21T22:58:29.907000", "content": "The truth is that after applying GST, the result gets worse. Train loss and val loss fluctuate in a small zone, it's really confused.
", "votes": 3, "replies": [ { "id": 2492222, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-10-22T10:35:13.763000", "content": "Hi @lizhecheng,
\nConsidering the limitation on GPU memory, I tried GST directly for layout. The learning curves generally look like what I posted above. After playing around with some hyperparameters, like stale embedding dropout, both training and validation losses decrease for nlp-random only. As for other metrics, all 4 layout collections have gradually improving OPAs. However, I’m still confused about the behavior of the training loss now. I’ll share more charts later.
", "votes": 1, "replies": [ { "id": 2492497, "author_name": "Zhecheng Li", "author_url": "", "post_date": "2023-10-22T15:18:45.127000", "content": "Thank you very much, we will continue exploring it.
", "votes": 1, "replies": [] }, { "id": 2492767, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-10-22T21:29:36.570000", "content": "Hi @lizhecheng,
\nI just tried another experiment to see the influence of modeling randomness (i.e., train models with another random seed with exactly the same validation setup). I chose layout-xla-default as an example. As can be seen, just changing the random seed led to a significantly different training process. Both training and validation losses decrease, though the training part seems within a narrower range. Also, kendall tau jumps about 1.5x. Hence, more robust CV scheme should be established. And, the training loss behavior still remains a mystery for me…
\n
So it means the only thing that we can do now is to change the random seed and expect a better result 😭
", "votes": 1, "replies": [] }, { "id": 2492917, "author_name": "Rob Freeman", "author_url": "", "post_date": "2023-10-23T03:08:26.380000", "content": "is it correct to assume that the amount of training data increased substantially as well? the loss value was two digits previously. or did you change your loss function as well?
", "votes": 2, "replies": [] }, { "id": 2492924, "author_name": "Vincent Sirius", "author_url": "", "post_date": "2023-10-23T03:19:06.197000", "content": "Can you be more specific? I don't really understand your comment.
", "votes": 2, "replies": [] }, { "id": 2493824, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-10-23T15:22:03.900000", "content": "Hi @lizhecheng,
\nI think the intrinsically small dataset is the problem leading to instability. The intention of sharing the charts is that I want to show the randomness matters a lot. I still try to find ways to mitigate the performance fluctuation (i.e., narrow the variance w.r.t. random seeds). However, I haven't had an answer now.
", "votes": 1, "replies": [] }, { "id": 2493830, "author_name": "Zhecheng Li", "author_url": "", "post_date": "2023-10-23T15:26:05.503000", "content": "Thank you so much. Will you good luck, we are also doing this now.
", "votes": 1, "replies": [] }, { "id": 2493852, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-10-23T15:50:02.760000", "content": "Hi @rob1080ti,
\nSorry for not clearly explaining the difference. The previous experiments with lower absolute loss values are trained with 32 sampled configurations per segment. In new experiments, I enlarge to 200 sampled configurations per segment. I don't change the loss function here.
\nAt least for me, the fluctuation of performance at different checkpoints is somewhat mitigated but not much. Hope this helps!
thank you! i have also tried more configurations which has helped but is expensive if trying to rapidly test other strategies in combination. one idea i had which may help the randomness issue is taking multiple checkpoints as topk checkpoints and running inference with these and try selecting based on the modeled overall time for all 1000 configs per file. if this was meaningful, it could also enable per file selections instead of per model checkpoint selections. how the config runtimes are scaled may or may not be important and regardless, this experiment may be very expensive and hence the topk checkpoints idea. also, it has occurred to me that dramatic changes in terms of configuration parameters may change absolute expected runtime costs so this may be very brittle if used across runs. (note: i have not tried this and i do not expect much time for further experimentation)
", "votes": 1, "replies": [] }, { "id": 2493884, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-10-23T16:11:59.133000", "content": "Hi @rdxsun,
\nSorry for confusing you. In the original post, I just ask if someone faces the similar issue as me (i.e., increasing training loss with decreasing val loss). The experimental settings are summarized above. What I forgot to mention was the number of sampled configurations used in Graph Segment Training, which was set to 32 configs for each segment with stale embedding dropout 0.5 in the original experiments.
\nAfter increasing the number of configs per segment to 200 in new experiments, I find out it helps stabilize the training process in terms of OPA (i.e., performance becomes less shaky). The potential reason behind the scene is that the small number of sampled configurations makes the training process too random to converge. Also, I believe better config sampling strategies can help (e.g., always considering the extreme runtime cases), which I'll try next.
\nAs for this thread, I want to share the important role the randomness plays here. Just changing the random seed can lead to significantly different performance. Hence, a more robust validation scheme should be established. Or, it's hard to trust what the validation set tells me, which might be caused by a merely lucky / unlucky seed.
", "votes": 1, "replies": [] }, { "id": 2493892, "author_name": "Zhecheng Li", "author_url": "", "post_date": "2023-10-23T16:16:21.517000", "content": "Sorry, my teammate wanted to reply to Rob, it was a mistake, we can understand what you shared. Thank you so much.🤣💪
", "votes": 1, "replies": [] }, { "id": 2493904, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-10-23T16:24:56.800000", "content": "Oh it's okay 🤣, I was just afraid that I didn't explain well. Good luck to you!!
", "votes": 1, "replies": [] }, { "id": 2493925, "author_name": "Zhecheng Li", "author_url": "", "post_date": "2023-10-23T16:32:30.910000", "content": "Your topic is really helpful and valuable, thanks so much. ❤️
", "votes": 2, "replies": [] }, { "id": 2493972, "author_name": "Rob Freeman", "author_url": "", "post_date": "2023-10-23T16:48:31.923000", "content": "Sorry, I wasn't sure if their question was directed at me. 😅
", "votes": 1, "replies": [] }, { "id": 2494004, "author_name": "Zhecheng Li", "author_url": "", "post_date": "2023-10-23T17:01:00.213000", "content": "Thanks for your great comments, now we have understood this is it correct to assume that the amount of training data increased substantially as well? the loss value was two digits previously. or did you change your loss function as well?
I think the opa/kt challenges are experienced across the board (see https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/443415 ). It mirrors my experience too, and I'm using a modified starter instead of the GST EFD repo.
\nSome thoughts: top teams may be cleaning the data, adding the additional features from the pb but not in the npz, combining the gnn with another approach, downsampling the training set (unlikely to improve opa/kt directly but may speed up iterating over experiments)
", "votes": 1, "replies": [ { "id": 2489919, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-10-20T09:45:29.993000", "content": "Hi @rob1080ti,
\nYeah, I also read that post and it seems that we're facing the similar challenges. Because I also had my own implementation of GST+EFD, I couldn't ensure that it's bug-free or not 😅. However, I drew some insights from another quick experiment. When I decrease dropout ratio in Stale Embedding Dropout from 0.5 to 0.25, the new learning curves are as follows,
\n
I'll share more observations if I find something interesting (and if time allows…). Also, thanks for sharing your thoughts.
", "votes": 1, "replies": [ { "id": 2490040, "author_name": "Rob Freeman", "author_url": "", "post_date": "2023-10-20T11:49:04.947000", "content": "i haven't really found a lot of benefit in tinkering with dropout. i'm also surprised by your training loss increase; i'm also using pairwise hinge for the layout models and training and validation losses both decrease for me. however, you may want to look at whether you are handling batching properly with the metrics (if you are using batching). i initially had some wacky results with kt before i realized that.
", "votes": 1, "replies": [] }, { "id": 2490553, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-10-20T19:23:02.610000", "content": "Hi @rob1080ti,
\nI'll check if there's any bug in my code, thanks a lot!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2489724, "author_name": "Amit Aharoni", "author_url": "", "post_date": "2023-10-20T07:04:39.263000", "content": "How does your train performance look like?
", "votes": 2, "replies": [ { "id": 2489923, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-10-20T09:47:19.143000", "content": "Hi @amitaharoni,
\nRunning evaluation on training set with different metrics is on my list and I'll share when experiments are done! Did you observe the similar behavior on the learning curve?
", "votes": 2, "replies": [ { "id": 2489931, "author_name": "Amit Aharoni", "author_url": "", "post_date": "2023-10-20T09:58:20.603000", "content": "To be honest I tried to run GST (all forms presented in the paper) and it just worsen my results. But I believe my implementation wasn't good. I might get back to it again later
", "votes": 2, "replies": [] }, { "id": 2490030, "author_name": "Rob Freeman", "author_url": "", "post_date": "2023-10-20T11:41:24.477000", "content": "btw for kt and opa i've noticed that approximations of each other look like kt=(opa-0.5)*2 and opa=(kt+1)/2
", "votes": 2, "replies": [] }, { "id": 2490568, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-10-20T19:42:16.873000", "content": "It's exactly the relationship between these two metrics. Let c be the number of concordant pairs and d the discordant ones. These two metrics can be formulated as follows,
\n$$OPA = \\frac{c}{c + d}$$
\n$$Kendall\\ \\tau = \\frac{c - d}{c + d}$$
Hence, the following relationship between the two holds,
\n$$OPA = \\frac{Kendall\\ \\tau + 1}{2}$$
May I ask how is the result now? Any improvement after using GST?
", "votes": 1, "replies": [] } ] } ] }, { "id": 2508380, "author_name": "MinhaoLin", "author_url": "", "post_date": "2023-11-01T16:37:23.793000", "content": "raise KeyError( \"Non-existent config key: f]\" . format(full key))
\nKeyError: ' Non-existent config key: train. mode
\nSorry i new to here.can i ask why my gst had this problem
", "votes": 0, "replies": [] }, { "id": 2492914, "author_name": "", "author_url": "", "post_date": "2023-10-23T03:07:35.193000", "content": "Hello,
\nAfter reviewing the TPUGRAPHS paper and your work, I'm somewhat perplexed. The paper emphasizes that by applying GST, it can address the issue of memory insufficiency while also improving accuracy relative to operations on full graphs. However, the content demonstrated in your charts seems to suggest the opposite outcome. Are there any potential explanations for this, or alternative implementation methods you might suggest?
\nAdditionally, I would like to inquire about your training model. If GST is not employed, what model do you utilize for training purposes?
\nThank you！
", "votes": 1, "replies": [ { "id": 2496996, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-10-24T11:42:16.073000", "content": "Hi @mstarlioning,
\nAs I mentioned above, considering the GPU memory limitation, I didn't even manage to implement full-graph training. My charts contain learning curves for 4 different types of layout collections, all of which are trained with GST framework. Hence, it's hard for me to comment on the comparison between full-graph training GST. But, maybe someone with sufficient hardware spec can help answer!
", "votes": 0, "replies": [ { "id": 2497183, "author_name": "Rob Freeman", "author_url": "", "post_date": "2023-10-24T13:56:56.220000", "content": "@abaojiang -- have you noticed a speedup with GPU training? With tfgnn, it is slower and a lot faster to just use CPU (this seemed to be the case with a 1080ti + i7 7700k).
", "votes": 1, "replies": [] }, { "id": 2497727, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-10-24T20:25:58.007000", "content": "Hi @rob1080ti,
\nBecause I need to balance the memory usage of CPU and GPU, I don't train models using CPU. Also, I don't use tfgnn framework, I rewrite the pipeline with pure torch. So, I think I can't comment on this topic.