![\"ouch\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1577010%2Fdff36be694a715752d6dcfe6ba4f8f03%2FScreenshot%20from%202023-11-16%2009-55-23.png?generation=1700088937258402&alt=media\")
When running experiments I kept noticing these huge spikes in the loss for certain graphs.
\n
Turned out this was being caused by the MSE component of the loss. Seemed like this is probably not good for the model.
\nI ran two experiments with the provided codebase: https://github.com/google-research-datasets/tpu_graphs
\nRMSE
\npython tiles_train.py --model=EarlyJoinSAGE --epochs=5 --losses='ListMLELoss:1,RMSE:0.02'
vs MSE
\npython tiles_train.py --model=EarlyJoinSAGE --epochs=5 --losses='ListMLELoss:1,MSE:0.02'
with metrics.py altered to add RMSE as
def root:\n return tf.sqrt(tf.reduce))\n\nLOSS_DICT = {\n 'ListMLELoss': tfr.keras.losses.,\n 'PairwiseHingeLoss': tfr.keras.losses.,\n 'MSE': tf.keras.losses.,\n 'RMSE': root_mean_squared_error\n}\nRMSE seems to increase the opa_metric and val_opa_metric significantly (and of course the loss goes down in both cases because the MSE component is now smaller)
![\"noice\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1577010%2Ff0100655072ce8611c8f73eb3ab5fd6f%2FScreenshot%20from%202023-11-16%2010-00-53.png?generation=1700089284457313&alt=media\")
maybe this is helpful to someone :D
", "messages": [ { "id": 2526561, "postDate": "2023-11-15T22:59:43.483Z", "content": "When running experiments I kept noticing these huge spikes in the loss for certain graphs.
\n
Turned out this was being caused by the MSE component of the loss. Seemed like this is probably not good for the model.
\nI ran two experiments with the provided codebase: https://github.com/google-research-datasets/tpu_graphs
\nRMSE
\npython tiles_train.py --model=EarlyJoinSAGE --epochs=5 --losses='ListMLELoss:1,RMSE:0.02'
vs MSE
\npython tiles_train.py --model=EarlyJoinSAGE --epochs=5 --losses='ListMLELoss:1,MSE:0.02'
with metrics.py altered to add RMSE as
def root:\n return tf.sqrt(tf.reduce))\n\nLOSS_DICT = {\n 'ListMLELoss': tfr.keras.losses.,\n 'PairwiseHingeLoss': tfr.keras.losses.,\n 'MSE': tf.keras.losses.,\n 'RMSE': root_mean_squared_error\n}\nRMSE seems to increase the opa_metric and val_opa_metric significantly (and of course the loss goes down in both cases because the MSE component is now smaller)
maybe this is helpful to someone :D
", "rawMarkdown": "When running experiments I kept noticing these huge spikes in the loss for certain graphs. \n\n\n\nTurned out this was being caused by the MSE component of the loss. Seemed like this is probably not good for the model. \n\nI ran two experiments with the provided codebase: https://github.com/google-research-datasets/tpu_graphs \n\nRMSE\n\n`python tiles_train.py --model=EarlyJoinSAGE --epochs=5 --losses='ListMLELoss:1,RMSE:0.02'`\n\nvs MSE\n\n`python tiles_train.py --model=EarlyJoinSAGE --epochs=5 --losses='ListMLELoss:1,MSE:0.02'`\n\nwith `metrics.py` altered to add RMSE as \n\n```\ndef root_mean_squared_error(y_true, y_pred):\n return tf.sqrt(tf.reduce_mean(tf.square(y_pred - y_true)))\n\nLOSS_DICT = {\n 'ListMLELoss': tfr.keras.losses.ListMLELoss(temperature=10),\n 'PairwiseHingeLoss': tfr.keras.losses.PairwiseHingeLoss(temperature=10),\n 'MSE': tf.keras.losses.MeanSquaredError(),\n 'RMSE': root_mean_squared_error\n}\n```\n\nRMSE seems to increase the `opa_metric` and `val_opa_metric` significantly (and of course the loss goes down in both cases because the MSE component is now smaller)\n\n\n\nmaybe this is helpful to someone :D", "votes": 6 } ], "comments": [], "raw_markdown_by_id": { "2526561": "When running experiments I kept noticing these huge spikes in the loss for certain graphs. \n\n\n\nTurned out this was being caused by the MSE component of the loss. Seemed like this is probably not good for the model. \n\nI ran two experiments with the provided codebase: https://github.com/google-research-datasets/tpu_graphs \n\nRMSE\n\n`python tiles_train.py --model=EarlyJoinSAGE --epochs=5 --losses='ListMLELoss:1,RMSE:0.02'`\n\nvs MSE\n\n`python tiles_train.py --model=EarlyJoinSAGE --epochs=5 --losses='ListMLELoss:1,MSE:0.02'`\n\nwith `metrics.py` altered to add RMSE as \n\n```\ndef root_mean_squared_error(y_true, y_pred):\n return tf.sqrt(tf.reduce_mean(tf.square(y_pred - y_true)))\n\nLOSS_DICT = {\n 'ListMLELoss': tfr.keras.losses.ListMLELoss(temperature=10),\n 'PairwiseHingeLoss': tfr.keras.losses.PairwiseHingeLoss(temperature=10),\n 'MSE': tf.keras.losses.MeanSquaredError(),\n 'RMSE': root_mean_squared_error\n}\n```\n\nRMSE seems to increase the `opa_metric` and `val_opa_metric` significantly (and of course the loss goes down in both cases because the MSE component is now smaller)\n\n\n\nmaybe this is helpful to someone :D" } }