Hi guys, I was experimenting with training on FP16 on a V100 GPU comparing the same model (4 layers of convolutions + max pool, followed by flattening and 3 layers of fully-connected layers including the final output layer) with full precision on this dataset with batch size of 96, and somehow FP16 have (either slightly or far more) worse performance than full precision. It seems like, with a learning rate finder, FP16 shows much more flat \"loss against learning rate\" curve than bumpy FP32. Training on FP16, two of the three times being tested shows 0.6-0.7 RocAuc training, and tweaking the learning rate (since it has a flat region over a large range of learning rate) can gives me about same accuracy as TF32, however, training for 10 epochs the RocAuc seems to fluctuate around that value of 0.82-0.84 without clear direction of improvement.
\nAnd yes, since the above has little experimentations one cannot be sure whether it is an illusion or solid. Perhaps there exists difference in improvements between different platforms (i.e. VM vs on Kaggle).
", "messages": [ { "id": "1393965", "postDate": "07/20/2021 04:07:44", "content": "Hi guys, I was experimenting with training on FP16 on a V100 GPU comparing the same model (4 layers of convolutions + max pool, followed by flattening and 3 layers of fully-connected layers including the final output layer) with full precision on this dataset with batch size of 96, and somehow FP16 have (either slightly or far more) worse performance than full precision. It seems like, with a learning rate finder, FP16 shows much more flat \"loss against learning rate\" curve than bumpy FP32. Training on FP16, two of the three times being tested shows 0.6-0.7 RocAuc training, and tweaking the learning rate (since it has a flat region over a large range of learning rate) can gives me about same accuracy as TF32, however, training for 10 epochs the RocAuc seems to fluctuate around that value of 0.82-0.84 without clear direction of improvement.
\nAnd yes, since the above has little experimentations one cannot be sure whether it is an illusion or solid. Perhaps there exists difference in improvements between different platforms (i.e. VM vs on Kaggle).
", "rawMarkdown": "Hi guys, I was experimenting with training on FP16 on a V100 GPU comparing the same model (4 layers of convolutions + max pool, followed by flattening and 3 layers of fully-connected layers including the final output layer) with full precision on [this](https://www.kaggle.com/wabinab/g2net-as-image) dataset with batch size of 96, and somehow FP16 have (either slightly or far more) worse performance than full precision. It seems like, with a learning rate finder, FP16 shows much more flat \"loss against learning rate\" curve than bumpy FP32. Training on FP16, two of the three times being tested shows 0.6-0.7 RocAuc training, and tweaking the learning rate (since it has a flat region over a large range of learning rate) can gives me about same accuracy as TF32, however, training for 10 epochs the RocAuc seems to fluctuate around that value of 0.82-0.84 without clear direction of improvement. \n\nAnd yes, since the above has little experimentations one cannot be sure whether it is an illusion or solid. Perhaps there exists difference in improvements between different platforms (i.e. VM vs on Kaggle).", "votes": null }, { "id": "1394493", "postDate": "07/20/2021 12:10:13", "content": "I think it is due to the effect of floating point. The value of the data in this competition is quite small (order ~ e-20), so when we change the data's dtype to FP16, they will not retain the original value.
\nThat can be confirmed by following simple code.
file_path = train[\"file_path\"][0]\nwaves = np.load(file_path).astype(np.float64)\nprint(waves)\n[[-5.94830548e-21 -5.84995448e-21 -5.42415169e-21 ... -6.06698987e-21\n -5.96345722e-21 -5.75778438e-21]\n [ 9.75407048e-22 4.52586118e-22 4.58643893e-23 ... -1.09608208e-20\n -1.09766636e-20 -1.10858129e-20]\n [-1.74871983e-21 -1.18286791e-21 -1.93223777e-21 ... 1.46502268e-21\n 2.18644864e-21 1.54085934e-21]]\nfile_path = train[\"file_path\"][0]\nwaves = np.load(file_path).astype(np.float32)\nprint(waves)\n[[-5.9483055e-21 -5.8499546e-21 -5.4241516e-21 ... -6.0669897e-21\n -5.9634572e-21 -5.7577845e-21]\n [ 9.7540710e-22 4.5258614e-22 4.5864389e-23 ... -1.0960821e-20\n -1.0976663e-20 -1.1085813e-20]\n [-1.7487198e-21 -1.1828679e-21 -1.9322378e-21 ... 1.4650227e-21\n 2.1864486e-21 1.5408594e-21]]\nfile_path = train[\"file_path\"][0]\nwaves = np.load(file_path).astype(np.float16)\nprint(waves)\n[[-0. -0. -0. ... -0. -0. -0.]\n [ 0. 0. 0. ... -0. -0. -0.]\n [-0. -0. -0. ... 0. 0. 0.]]\nAs @naoism pointed out, FP16 cannot store extremely small values in the order of e-20 such as the data from this competition (this is because FP16 has only 5 bits for exponent!).
\nBut I don't think your case is related to the wave data because at least you were able to reach score of 0,8 or above.
\nI recommend you partially disable AMP autocast by adding a with torch.cuda.amp.autocast(enabled=False): subregion to your model, and see what happens. I noticed nnAudio package sometimes return NaN with AMP enabled.
\nHope this helps you!
Thank you for your comment. Indeed, it seems that wave data may be not relevant if the score of 0.8 was obtained.
", "rawMarkdown": "Thank you for your comment. Indeed, it seems that wave data may be not relevant if the score of 0.8 was obtained.", "votes": null }, { "id": "1394751", "postDate": "07/20/2021 15:16:26", "content": "it is better to use mixed precision, for instance torch.cuda.amp if you use pytorch. With this, model weights are FP32 but gradient is computed using FP16. Moreover amp rescales gradient so that max FP16 precision is kept.
\nI train all my models with amp now.
", "rawMarkdown": "it is better to use mixed precision, for instance torch.cuda.amp if you use pytorch. With this, model weights are FP32 but gradient is computed using FP16. Moreover amp rescales gradient so that max FP16 precision is kept.\n\nI train all my models with amp now.", "votes": null }, { "id": "1394752", "postDate": "07/20/2021 15:20:41", "content": "I disagree, it is dangerous to disable autocast, as you're more likely to get NaN or wrong gradients.
", "rawMarkdown": "I disagree, it is dangerous to disable autocast, as you're more likely to get NaN or wrong gradients.", "votes": null }, { "id": "1395138", "postDate": "07/21/2021 00:22:02", "content": "I assumed @wabinab uses precision instead of FP16 only, thus partially disabling autocast (=forcing FP32 operation) is a good option for debugging. I suppose this behaviour is due to autocast's failure to determine which cast to use for some specific operations used in nnAudio package.
\nI trained all my models with amp (autocast partially disabled).
Thanks for your comment @analokamus
\nOne uses fastai's \"fit_one_cycle\" to do the training so underlyingly one haven't check the source code of which part it uses autocast. One have been experimenting it with pretrained efficientnet one could still get 0.8+ AUC sometimes though one have difficulty to push it higher than 0.85 yet. It might be perhaps tuning learning rate (and maybe other hyperparameters but one haven't tried it out) is quite important.
\nAs for @naoism yes in this case all the data are represented as png images so they're scaled to either 0-255 or 0-1 depending on convention.
", "rawMarkdown": "Thanks for your comment @analokamus \n\nOne uses fastai's \"fit_one_cycle\" to do the training so underlyingly one haven't check the source code of which part it uses autocast. One have been experimenting it with pretrained efficientnet one could still get 0.8+ AUC sometimes though one have difficulty to push it higher than 0.85 yet. It might be perhaps tuning learning rate (and maybe other hyperparameters but one haven't tried it out) is quite important. \n\nAs for @naoism yes in this case all the data are represented as png images so they're scaled to either 0-255 or 0-1 depending on convention.", "votes": null }, { "id": "1395526", "postDate": "07/21/2021 09:35:35", "content": "@analokamus Fair enough, I thought you were discussing Scaler, sorry.
", "rawMarkdown": "analokamus Fair enough, I thought you were discussing Scaler, sorry.", "votes": null }, { "id": "1395667", "postDate": "07/21/2021 12:43:13", "content": "Good day
\nI read somewhere that the neural networks needed is one input layer two convolutions ReLu and the output…
\nI believe it was the MatLab model(?)…
\nThen add Propagation from two convolution networks and that’s what gives you your probability is using Bayesian theory
\nAlso, tensor flow can be used (?)
Good day, the FP16 I read could not be used in this as it causes issues of overfitting So using FP64 is the one to use I read this on the MATLAB documents for this very first detection of a GWbiH
\nhttps://www.mathworks.com/company/newsletters/articles/confirming-the-first-ever-detection-of-gravitational-waves-by-analyzing-laser-interferometer-data.html
Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
I think it is due to the effect of floating point. The value of the data in this competition is quite small (order ~ e-20), so when we change the data's dtype to FP16, they will not retain the original value.
\nThat can be confirmed by following simple code.
file_path = train[\"file_path\"][0]\nwaves = np.load(file_path).astype(np.float64)\nprint(waves)\n[[-5.94830548e-21 -5.84995448e-21 -5.42415169e-21 ... -6.06698987e-21\n -5.96345722e-21 -5.75778438e-21]\n [ 9.75407048e-22 4.52586118e-22 4.58643893e-23 ... -1.09608208e-20\n -1.09766636e-20 -1.10858129e-20]\n [-1.74871983e-21 -1.18286791e-21 -1.93223777e-21 ... 1.46502268e-21\n 2.18644864e-21 1.54085934e-21]]\nfile_path = train[\"file_path\"][0]\nwaves = np.load(file_path).astype(np.float32)\nprint(waves)\n[[-5.9483055e-21 -5.8499546e-21 -5.4241516e-21 ... -6.0669897e-21\n -5.9634572e-21 -5.7577845e-21]\n [ 9.7540710e-22 4.5258614e-22 4.5864389e-23 ... -1.0960821e-20\n -1.0976663e-20 -1.1085813e-20]\n [-1.7487198e-21 -1.1828679e-21 -1.9322378e-21 ... 1.4650227e-21\n 2.1864486e-21 1.5408594e-21]]\nfile_path = train[\"file_path\"][0]\nwaves = np.load(file_path).astype(np.float16)\nprint(waves)\n[[-0. -0. -0. ... -0. -0. -0.]\n [ 0. 0. 0. ... -0. -0. -0.]\n [-0. -0. -0. ... 0. 0. 0.]]\nAs @naoism pointed out, FP16 cannot store extremely small values in the order of e-20 such as the data from this competition (this is because FP16 has only 5 bits for exponent!).
\nBut I don't think your case is related to the wave data because at least you were able to reach score of 0,8 or above.
\nI recommend you partially disable AMP autocast by adding a with torch.cuda.amp.autocast(enabled=False): subregion to your model, and see what happens. I noticed nnAudio package sometimes return NaN with AMP enabled.
\nHope this helps you!
Thank you for your comment. Indeed, it seems that wave data may be not relevant if the score of 0.8 was obtained.
", "votes": null, "replies": [] }, { "id": 1394752, "author_name": "cpmpml", "author_url": "", "post_date": "07/20/2021 15:20:41", "content": "I disagree, it is dangerous to disable autocast, as you're more likely to get NaN or wrong gradients.
", "votes": null, "replies": [] }, { "id": 1395138, "author_name": "analokamus", "author_url": "", "post_date": "07/21/2021 00:22:02", "content": "I assumed @wabinab uses precision instead of FP16 only, thus partially disabling autocast (=forcing FP32 operation) is a good option for debugging. I suppose this behaviour is due to autocast's failure to determine which cast to use for some specific operations used in nnAudio package.
\nI trained all my models with amp (autocast partially disabled).
Thanks for your comment @analokamus
\nOne uses fastai's \"fit_one_cycle\" to do the training so underlyingly one haven't check the source code of which part it uses autocast. One have been experimenting it with pretrained efficientnet one could still get 0.8+ AUC sometimes though one have difficulty to push it higher than 0.85 yet. It might be perhaps tuning learning rate (and maybe other hyperparameters but one haven't tried it out) is quite important.
\nAs for @naoism yes in this case all the data are represented as png images so they're scaled to either 0-255 or 0-1 depending on convention.
", "votes": null, "replies": [] }, { "id": 1395526, "author_name": "cpmpml", "author_url": "", "post_date": "07/21/2021 09:35:35", "content": "@analokamus Fair enough, I thought you were discussing Scaler, sorry.
", "votes": null, "replies": [] } ] }, { "id": 1394751, "author_name": "cpmpml", "author_url": "", "post_date": "07/20/2021 15:16:26", "content": "it is better to use mixed precision, for instance torch.cuda.amp if you use pytorch. With this, model weights are FP32 but gradient is computed using FP16. Moreover amp rescales gradient so that max FP16 precision is kept.
\nI train all my models with amp now.
", "votes": null, "replies": [] }, { "id": 1395667, "author_name": "captbullett", "author_url": "", "post_date": "07/21/2021 12:43:13", "content": "Good day
\nI read somewhere that the neural networks needed is one input layer two convolutions ReLu and the output…
\nI believe it was the MatLab model(?)…
\nThen add Propagation from two convolution networks and that’s what gives you your probability is using Bayesian theory
\nAlso, tensor flow can be used (?)
Good day, the FP16 I read could not be used in this as it causes issues of overfitting So using FP64 is the one to use I read this on the MATLAB documents for this very first detection of a GWbiH
\nhttps://www.mathworks.com/company/newsletters/articles/confirming-the-first-ever-detection-of-gravitational-waves-by-analyzing-laser-interferometer-data.html
Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris