Hey folks,
\nI have been working on some GNN approaches for several days, and I can't find why they don't converge.
\nI haven't seen any GNN public notebooks yet, except the ones based on GraphNet, so I have this hope that I am not the only one struggling with GNNs. Do you have any experience to share? Any advice to make models learn would be highly appreciated. So far, all it predicts is garbage.
Hey folks,
\nI have been working on some GNN approaches for several days, and I can't find why they don't converge.
\nI haven't seen any GNN public notebooks yet, except the ones based on GraphNet, so I have this hope that I am not the only one struggling with GNNs. Do you have any experience to share? Any advice to make models learn would be highly appreciated. So far, all it predicts is garbage.
It can take a while for the model to start converging. Typically mine will predict an \"average\" result for about 15 epochs on a single batch training set before it begins to show any improvement.
", "rawMarkdown": "It can take a while for the model to start converging. Typically mine will predict an \"average\" result for about 15 epochs on a single batch training set before it begins to show any improvement.", "votes": null }, { "id": "2134711", "postDate": "02/08/2023 07:57:10", "content": "So yours randomly predicts during 15 epochs, and then converges ?
\nWhich LR/opt/loss do you use ? and did you switch to a classif task or did you stick to regression ?
It does show some hint that it's converging. During those 15 epochs the loss does improve a tiny bit each validation, somewhere around the order of 1e-6.
\nI started with the hyperparameters in the graphnet paper and was getting that behavior for regression. It should just be the LR/opt/loss released in the graphnet kernel. Only difference is that I changed my batch size to 1024.
", "rawMarkdown": "It does show some hint that it's converging. During those 15 epochs the loss does improve a tiny bit each validation, somewhere around the order of 1e-6.\n\nI started with the hyperparameters in the graphnet paper and was getting that behavior for regression. It should just be the LR/opt/loss released in the graphnet kernel. Only difference is that I changed my batch size to 1024.", "votes": null }, { "id": "2146085", "postDate": "02/15/2023 15:35:59", "content": "Definitely Agree that there doesn't seem to be much content on GNNs (especially for regression) out there! Would love to see anything anyone has found that's helpful.
", "rawMarkdown": "Definitely Agree that there doesn't seem to be much content on GNNs (especially for regression) out there! Would love to see anything anyone has found that's helpful.", "votes": null }, { "id": "2149005", "postDate": "02/17/2023 20:31:11", "content": "I was training a CNN/LSTM and had issues converging when using a high learning rate (>1e-4) or any kind of strong regularization.
", "rawMarkdown": "I was training a CNN/LSTM and had issues converging when using a high learning rate (>1e-4) or any kind of strong regularization.", "votes": null } ], "comments": [ { "id": 2134331, "author_name": "bsmit1659", "author_url": "", "post_date": "02/07/2023 22:35:32", "content": "It can take a while for the model to start converging. Typically mine will predict an \"average\" result for about 15 epochs on a single batch training set before it begins to show any improvement.
", "votes": null, "replies": [ { "id": 2134711, "author_name": "louisstefanuto", "author_url": "", "post_date": "02/08/2023 07:57:10", "content": "So yours randomly predicts during 15 epochs, and then converges ?
\nWhich LR/opt/loss do you use ? and did you switch to a classif task or did you stick to regression ?
It does show some hint that it's converging. During those 15 epochs the loss does improve a tiny bit each validation, somewhere around the order of 1e-6.
\nI started with the hyperparameters in the graphnet paper and was getting that behavior for regression. It should just be the LR/opt/loss released in the graphnet kernel. Only difference is that I changed my batch size to 1024.
", "votes": null, "replies": [] } ] } ] }, { "id": 2146085, "author_name": "rhysie", "author_url": "", "post_date": "02/15/2023 15:35:59", "content": "Definitely Agree that there doesn't seem to be much content on GNNs (especially for regression) out there! Would love to see anything anyone has found that's helpful.
", "votes": null, "replies": [] }, { "id": 2149005, "author_name": "tatelarkin", "author_url": "", "post_date": "02/17/2023 20:31:11", "content": "I was training a CNN/LSTM and had issues converging when using a high learning rate (>1e-4) or any kind of strong regularization.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2134275": "Hey folks,\nI have been working on some GNN approaches for several days, and I can't find why they don't converge.\nI haven't seen any GNN public notebooks yet, except the ones based on GraphNet, so I have this hope that I am not the only one struggling with GNNs. Do you have any experience to share? Any advice to make models learn would be highly appreciated. So far, all it predicts is garbage.", "2134331": "It can take a while for the model to start converging. Typically mine will predict an \"average\" result for about 15 epochs on a single batch training set before it begins to show any improvement.", "2134711": "So yours randomly predicts during 15 epochs, and then converges ?\nWhich LR/opt/loss do you use ? and did you switch to a classif task or did you stick to regression ?", "2134981": "It does show some hint that it's converging. During those 15 epochs the loss does improve a tiny bit each validation, somewhere around the order of 1e-6.\n\nI started with the hyperparameters in the graphnet paper and was getting that behavior for regression. It should just be the LR/opt/loss released in the graphnet kernel. Only difference is that I changed my batch size to 1024.", "2146085": "Definitely Agree that there doesn't seem to be much content on GNNs (especially for regression) out there! Would love to see anything anyone has found that's helpful.", "2149005": "I was training a CNN/LSTM and had issues converging when using a high learning rate (>1e-4) or any kind of strong regularization." }, "source": "meta" }