\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F19099%2Fac63fa4bb56c678383c0b68a163c5f4e%2Factivation.png?generation=1665783746972764&alt=media\")
Hi Kaggle fellows,
\nIn this thread I would like to discuss the differences in the NN activation functions, as currently we have the top-1 kernel based on the NN built with FastAI framework.
\nWhen you try to create the better NN model, you almost always try to find the best NN structure you can - and this is not only the number of layers and their sizes! It's also important to choose the right activation for the layers to prevent the usual NN problems like vanishing or exploding gradients, bad training speed or final quality.
\nWhat activations do we know, how do they look like in comparison to each other?
\n
The most commonly known variants are Sigmoid, Tanh and ReLU, but all of them have some advantages and disadvantages. Let's take a closer look for the ReLU, the most common default activation for the frameworks nowadays.
\nAdvantages of ReLU:
\nx > 0 ReLU has constant gradient, which reduces the chance of vanishing gradients at any point of time (results in faster learning)x <= 0 the gradient is zero and hence less number of neurons will be fired which reduce overfitting and is also cost efficientDisadvantage of ReLU - the dying ReLU problem. A ReLU neuron is dead if it is stuck in the negative side and always outputs zero (so it is impossible for the neuron to recover back)
\nTo fix the ReLU disadvantage in 2017 Google Brain invented the Swish activation - almost ReLU, but with fixed negative part and the parameter to change the slope for the positives. It shows better scores on the different benchmarks like ImageNet and everybody was happy about that.
But still researchers wanted better accuracy, stability and universality so in 2019 the Mish activation appears. Why it can be better than Swish?
Mish outperforms in case of noisy input conditions in comparison with ReLU and SwishMish compared with SwishMish has shown to have a consistent improvement over Swish using different dropout ratesMish is also usually better than SwishSo that's why Mish become the better, more stable and universal activation for NNs but it is also important to say - to find the optimal activation function for the specific ML task/competition, you need to check different variants and compare them on your validation set. The data and the experiment should say what is the best choice, not the papers.
Hope this helps you to make a better choice :)
\nAlex
\nP.S. For more details and graphs you can check this link
", "messages": [ { "id": "1987904", "postDate": "10/14/2022 22:13:10", "content": "Hi Kaggle fellows,
\nIn this thread I would like to discuss the differences in the NN activation functions, as currently we have the top-1 kernel based on the NN built with FastAI framework.
\nWhen you try to create the better NN model, you almost always try to find the best NN structure you can - and this is not only the number of layers and their sizes! It's also important to choose the right activation for the layers to prevent the usual NN problems like vanishing or exploding gradients, bad training speed or final quality.
\nWhat activations do we know, how do they look like in comparison to each other?
\n
The most commonly known variants are Sigmoid, Tanh and ReLU, but all of them have some advantages and disadvantages. Let's take a closer look for the ReLU, the most common default activation for the frameworks nowadays.
\nAdvantages of ReLU:
\nx > 0 ReLU has constant gradient, which reduces the chance of vanishing gradients at any point of time (results in faster learning)x <= 0 the gradient is zero and hence less number of neurons will be fired which reduce overfitting and is also cost efficientDisadvantage of ReLU - the dying ReLU problem. A ReLU neuron is dead if it is stuck in the negative side and always outputs zero (so it is impossible for the neuron to recover back)
\nTo fix the ReLU disadvantage in 2017 Google Brain invented the Swish activation - almost ReLU, but with fixed negative part and the parameter to change the slope for the positives. It shows better scores on the different benchmarks like ImageNet and everybody was happy about that.
But still researchers wanted better accuracy, stability and universality so in 2019 the Mish activation appears. Why it can be better than Swish?
Mish outperforms in case of noisy input conditions in comparison with ReLU and SwishMish compared with SwishMish has shown to have a consistent improvement over Swish using different dropout ratesMish is also usually better than SwishSo that's why Mish become the better, more stable and universal activation for NNs but it is also important to say - to find the optimal activation function for the specific ML task/competition, you need to check different variants and compare them on your validation set. The data and the experiment should say what is the best choice, not the papers.
Hope this helps you to make a better choice :)
\nAlex
\nP.S. For more details and graphs you can check this link
", "rawMarkdown": "Hi Kaggle fellows, \n\nIn this thread I would like to discuss the differences in the NN activation functions, as currently we have [the top-1 kernel](https://www.kaggle.com/code/alexryzhkov/tps-2022-10-fastai-with-multistart-and-tta) based on the NN built with FastAI framework.\n\nWhen you try to create the better NN model, you almost always try to find the best NN structure you can - and this is not only the number of layers and their sizes! It's also **important to choose the right activation for the layers** to prevent the usual NN problems like vanishing or exploding gradients, bad training speed or final quality. \n\nWhat activations do we know, how do they look like in comparison to each other?\n \n\nThe most commonly known variants are Sigmoid, Tanh and ReLU, but all of them have some advantages and disadvantages. Let's take a closer look for the ReLU, the most common default activation for the frameworks nowadays.\n\n**Advantages of ReLU:**\n- For `x > 0` ReLU has constant gradient, which reduces the chance of vanishing gradients at any point of time (results in faster learning)\n- For `x <= 0` the gradient is zero and hence less number of neurons will be fired which reduce overfitting and is also cost efficient\n- ReLU has better convergence performance as compared to other activation functions\n\n**Disadvantage of ReLU** - the dying ReLU problem. A ReLU neuron is dead if it is stuck in the negative side and always outputs zero (so it is impossible for the neuron to recover back)\n\n**To fix the ReLU disadvantage** in 2017 Google Brain invented the `Swish` activation - almost `ReLU`, but with fixed negative part and the parameter to change the slope for the positives. It shows better scores on the different benchmarks like ImageNet and everybody was happy about that.\n\nBut still researchers wanted **better accuracy, stability and universality** so in 2019 the `Mish` activation appears. Why it can be better than `Swish`?\n- `Mish` outperforms in case of noisy input conditions in comparison with `ReLU` and `Swish`\n- Test accuracy with different optimizers has less drop in case of `Mish` compared with `Swish`\n- `Mish` has shown to have a consistent improvement over `Swish` using different dropout rates\n- Test accuracy with different learning rates for `Mish` is also usually better than `Swish`\n\nSo that's why `Mish` become the better, more stable and universal activation for NNs but it is also important to say - **to find the optimal activation function for the specific ML task/competition, you need to check different variants and compare them on your validation set**. The data and the experiment should say what is the best choice, not the papers.\n\nHope this helps you to make a better choice :)\n\nAlex\n\nP.S. For more details and graphs you can check [this link](https://krutikabapat.github.io/Swish-Vs-Mish-Latest-Activation-Functions/)", "votes": null }, { "id": "1988514", "postDate": "10/15/2022 11:10:11", "content": "Great post! In my experience, the choice of activation functions don't have a a huge impact for shallow neural networks (i.e. NNs with a small number of layers, e.g. less than 4) but it becomes very important for ultra deep neural networks (50+ layers) where you have the problem of exploding and vanishing gradients. I haven't heard of mish before though so thanks for sharing!
", "rawMarkdown": "Great post! In my experience, the choice of activation functions don't have a a huge impact for shallow neural networks (i.e. NNs with a small number of layers, e.g. less than 4) but it becomes very important for ultra deep neural networks (50+ layers) where you have the problem of exploding and vanishing gradients. I haven't heard of mish before though so thanks for sharing!", "votes": null }, { "id": "1991099", "postDate": "10/17/2022 01:25:19", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "1991327", "postDate": "10/17/2022 05:28:20", "content": "Good post! It helps me a lot.
", "rawMarkdown": "Good post! It helps me a lot.", "votes": null } ], "comments": [ { "id": 1988514, "author_name": "samuelcortinhas", "author_url": "", "post_date": "10/15/2022 11:10:11", "content": "Great post! In my experience, the choice of activation functions don't have a a huge impact for shallow neural networks (i.e. NNs with a small number of layers, e.g. less than 4) but it becomes very important for ultra deep neural networks (50+ layers) where you have the problem of exploding and vanishing gradients. I haven't heard of mish before though so thanks for sharing!
", "votes": null, "replies": [] }, { "id": 1991099, "author_name": "pomiro", "author_url": "", "post_date": "10/17/2022 01:25:19", "content": "Thank you!
", "votes": null, "replies": [] }, { "id": 1991327, "author_name": "zhuoranjordan", "author_url": "", "post_date": "10/17/2022 05:28:20", "content": "Good post! It helps me a lot.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1987904": "Hi Kaggle fellows, \n\nIn this thread I would like to discuss the differences in the NN activation functions, as currently we have [the top-1 kernel](https://www.kaggle.com/code/alexryzhkov/tps-2022-10-fastai-with-multistart-and-tta) based on the NN built with FastAI framework.\n\nWhen you try to create the better NN model, you almost always try to find the best NN structure you can - and this is not only the number of layers and their sizes! It's also **important to choose the right activation for the layers** to prevent the usual NN problems like vanishing or exploding gradients, bad training speed or final quality. \n\nWhat activations do we know, how do they look like in comparison to each other?\n \n\nThe most commonly known variants are Sigmoid, Tanh and ReLU, but all of them have some advantages and disadvantages. Let's take a closer look for the ReLU, the most common default activation for the frameworks nowadays.\n\n**Advantages of ReLU:**\n- For `x > 0` ReLU has constant gradient, which reduces the chance of vanishing gradients at any point of time (results in faster learning)\n- For `x <= 0` the gradient is zero and hence less number of neurons will be fired which reduce overfitting and is also cost efficient\n- ReLU has better convergence performance as compared to other activation functions\n\n**Disadvantage of ReLU** - the dying ReLU problem. A ReLU neuron is dead if it is stuck in the negative side and always outputs zero (so it is impossible for the neuron to recover back)\n\n**To fix the ReLU disadvantage** in 2017 Google Brain invented the `Swish` activation - almost `ReLU`, but with fixed negative part and the parameter to change the slope for the positives. It shows better scores on the different benchmarks like ImageNet and everybody was happy about that.\n\nBut still researchers wanted **better accuracy, stability and universality** so in 2019 the `Mish` activation appears. Why it can be better than `Swish`?\n- `Mish` outperforms in case of noisy input conditions in comparison with `ReLU` and `Swish`\n- Test accuracy with different optimizers has less drop in case of `Mish` compared with `Swish`\n- `Mish` has shown to have a consistent improvement over `Swish` using different dropout rates\n- Test accuracy with different learning rates for `Mish` is also usually better than `Swish`\n\nSo that's why `Mish` become the better, more stable and universal activation for NNs but it is also important to say - **to find the optimal activation function for the specific ML task/competition, you need to check different variants and compare them on your validation set**. The data and the experiment should say what is the best choice, not the papers.\n\nHope this helps you to make a better choice :)\n\nAlex\n\nP.S. For more details and graphs you can check [this link](https://krutikabapat.github.io/Swish-Vs-Mish-Latest-Activation-Functions/)", "1988514": "Great post! In my experience, the choice of activation functions don't have a a huge impact for shallow neural networks (i.e. NNs with a small number of layers, e.g. less than 4) but it becomes very important for ultra deep neural networks (50+ layers) where you have the problem of exploding and vanishing gradients. I haven't heard of mish before though so thanks for sharing!", "1991099": "Thank you!", "1991327": "Good post! It helps me a lot." }, "source": "meta" }