I heard a lot about vanishing gradient being a problem during model training especially when the neural network is deep, but I've never seen the problem. Has anyone experienced it in practice? What's the effect?
", "messages": [ { "id": "529185", "postDate": "05/09/2019 11:15:52", "content": "I heard a lot about vanishing gradient being a problem during model training especially when the neural network is deep, but I've never seen the problem. Has anyone experienced it in practice? What's the effect?
", "rawMarkdown": "I heard a lot about vanishing gradient being a problem during model training especially when the neural network is deep, but I've never seen the problem. Has anyone experienced it in practice? What's the effect?", "votes": null }, { "id": "529199", "postDate": "05/09/2019 12:04:43", "content": "It happened mainly when sigmoid or tanh were the activation functions. If you use relu, then vanishing gradient becomes a problem only if you use very deep networks and don't initialize weights the right way. See for some explanations: https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79
\n\nNote that exploding gradient can now be a problem. It was not a problem with sigmoid or tanh activations.
", "rawMarkdown": "It happened mainly when sigmoid or tanh were the activation functions. If you use relu, then vanishing gradient becomes a problem only if you use very deep networks and don't initialize weights the right way. See for some explanations: https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79\n\nNote that exploding gradient can now be a problem. It was not a problem with sigmoid or tanh activations.", "votes": null }, { "id": "529218", "postDate": "05/09/2019 13:06:08", "content": "Thanks for answering! How do you identify during training that vanishing(or exploding) gradient is happening? Is it by seeing that loss remains high and doesn't converge?
", "rawMarkdown": "Thanks for answering! How do you identify during training that vanishing(or exploding) gradient is happening? Is it by seeing that loss remains high and doesn't converge?", "votes": null }, { "id": "529256", "postDate": "05/09/2019 14:27:19", "content": "Hi @vivaroma, \nVanishing gradient means that the gradients of earlier layers are very small. Therefore, the corresponding weights barely move after the update of backpropagation. \nYou can identify this problem by looking at the gradients with respect to the weights of earlier layers. If they are very small then you have a vanishing gradient.
", "rawMarkdown": "Hi @vivaroma, \nVanishing gradient means that the gradients of earlier layers are very small. Therefore, the corresponding weights barely move after the update of backpropagation. \nYou can identify this problem by looking at the gradients with respect to the weights of earlier layers. If they are very small then you have a vanishing gradient.", "votes": null }, { "id": "529263", "postDate": "05/09/2019 14:37:16", "content": "As said, it should not happen if you use small number of layers. Also, recent CNN architectures like resnet pass residual info from layers to layers, eliminating in practice vanishing gradient. For exploding gradient, you see if if loss suddenly explodes.
", "rawMarkdown": "As said, it should not happen if you use small number of layers. Also, recent CNN architectures like resnet pass residual info from layers to layers, eliminating in practice vanishing gradient. For exploding gradient, you see if if loss suddenly explodes.", "votes": null }, { "id": "529351", "postDate": "05/09/2019 17:39:19", "content": "@vivaroma, a small gradient means that the weights and biases of the initial layers will not be updated effectively with each training session. Since these initial layers are often crucial to recognizing the core elements of the input data, it can lead to overall inaccuracy of the whole network.
", "rawMarkdown": "vivaroma, a small gradient means that the weights and biases of the initial layers will not be updated effectively with each training session. Since these initial layers are often crucial to recognizing the core elements of the input data, it can lead to overall inaccuracy of the whole network.", "votes": null }, { "id": "529443", "postDate": "05/09/2019 22:50:43", "content": "I see...Do you actively check the gradients in practice to avoid vanishing gradient? Sounds like it only happens under certain conditions/architectures, and we might not need to worry about it otherwise.
", "rawMarkdown": "I see...Do you actively check the gradients in practice to avoid vanishing gradient? Sounds like it only happens under certain conditions/architectures, and we might not need to worry about it otherwise.", "votes": null }, { "id": "529666", "postDate": "05/10/2019 13:14:19", "content": "As everyone else said. I'll also add that vanishing gradient affected specially Vanilla RNN because propagating the error along the time made the gradient to become exponentially small for long time sequences. LSTM and GRU solves this issue teorically but on practice you have to be carefull because with some really long sequences you will most probably have vanishing gradient.
\n\nIf you are interested more on this topic I'll suggest reading this post which explains everything really well.
", "rawMarkdown": "As everyone else said. I'll also add that vanishing gradient affected specially Vanilla RNN because propagating the error along the time made the gradient to become exponentially small for long time sequences. LSTM and GRU solves this issue teorically but on practice you have to be carefull because with some really long sequences you will most probably have vanishing gradient.\n\nIf you are interested more on this topic I'll suggest reading this [post](https://medium.com/datadriveninvestor/how-do-lstm-networks-solve-the-problem-of-vanishing-gradients-a6784971a577) which explains everything really well.", "votes": null }, { "id": "531785", "postDate": "05/15/2019 14:37:09", "content": "How do you deal with exploding gradient then?
", "rawMarkdown": "How do you deal with exploding gradient then?", "votes": null }, { "id": "531798", "postDate": "05/15/2019 15:02:58", "content": "lower learning rate, clip gradient, check sample weights, ...
", "rawMarkdown": "lower learning rate, clip gradient, check sample weights, ...", "votes": null } ], "comments": [ { "id": 529199, "author_name": "cpmpml", "author_url": "", "post_date": "05/09/2019 12:04:43", "content": "It happened mainly when sigmoid or tanh were the activation functions. If you use relu, then vanishing gradient becomes a problem only if you use very deep networks and don't initialize weights the right way. See for some explanations: https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79
\n\nNote that exploding gradient can now be a problem. It was not a problem with sigmoid or tanh activations.
", "votes": null, "replies": [ { "id": 529218, "author_name": "vivaroma", "author_url": "", "post_date": "05/09/2019 13:06:08", "content": "Thanks for answering! How do you identify during training that vanishing(or exploding) gradient is happening? Is it by seeing that loss remains high and doesn't converge?
", "votes": null, "replies": [] }, { "id": 529263, "author_name": "cpmpml", "author_url": "", "post_date": "05/09/2019 14:37:16", "content": "As said, it should not happen if you use small number of layers. Also, recent CNN architectures like resnet pass residual info from layers to layers, eliminating in practice vanishing gradient. For exploding gradient, you see if if loss suddenly explodes.
", "votes": null, "replies": [] }, { "id": 531785, "author_name": "vivaroma", "author_url": "", "post_date": "05/15/2019 14:37:09", "content": "How do you deal with exploding gradient then?
", "votes": null, "replies": [] }, { "id": 531798, "author_name": "cpmpml", "author_url": "", "post_date": "05/15/2019 15:02:58", "content": "lower learning rate, clip gradient, check sample weights, ...
", "votes": null, "replies": [] } ] }, { "id": 529256, "author_name": "adnanemiri", "author_url": "", "post_date": "05/09/2019 14:27:19", "content": "Hi @vivaroma, \nVanishing gradient means that the gradients of earlier layers are very small. Therefore, the corresponding weights barely move after the update of backpropagation. \nYou can identify this problem by looking at the gradients with respect to the weights of earlier layers. If they are very small then you have a vanishing gradient.
", "votes": null, "replies": [ { "id": 529443, "author_name": "vivaroma", "author_url": "", "post_date": "05/09/2019 22:50:43", "content": "I see...Do you actively check the gradients in practice to avoid vanishing gradient? Sounds like it only happens under certain conditions/architectures, and we might not need to worry about it otherwise.
", "votes": null, "replies": [] } ] }, { "id": 529351, "author_name": "makinghistory", "author_url": "", "post_date": "05/09/2019 17:39:19", "content": "@vivaroma, a small gradient means that the weights and biases of the initial layers will not be updated effectively with each training session. Since these initial layers are often crucial to recognizing the core elements of the input data, it can lead to overall inaccuracy of the whole network.
", "votes": null, "replies": [] }, { "id": 529666, "author_name": "guillemdelgado", "author_url": "", "post_date": "05/10/2019 13:14:19", "content": "As everyone else said. I'll also add that vanishing gradient affected specially Vanilla RNN because propagating the error along the time made the gradient to become exponentially small for long time sequences. LSTM and GRU solves this issue teorically but on practice you have to be carefull because with some really long sequences you will most probably have vanishing gradient.
\n\nIf you are interested more on this topic I'll suggest reading this post which explains everything really well.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "529185": "I heard a lot about vanishing gradient being a problem during model training especially when the neural network is deep, but I've never seen the problem. Has anyone experienced it in practice? What's the effect?", "529199": "It happened mainly when sigmoid or tanh were the activation functions. If you use relu, then vanishing gradient becomes a problem only if you use very deep networks and don't initialize weights the right way. See for some explanations: https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79\n\nNote that exploding gradient can now be a problem. It was not a problem with sigmoid or tanh activations.", "529218": "Thanks for answering! How do you identify during training that vanishing(or exploding) gradient is happening? Is it by seeing that loss remains high and doesn't converge?", "529256": "Hi @vivaroma, \nVanishing gradient means that the gradients of earlier layers are very small. Therefore, the corresponding weights barely move after the update of backpropagation. \nYou can identify this problem by looking at the gradients with respect to the weights of earlier layers. If they are very small then you have a vanishing gradient.", "529263": "As said, it should not happen if you use small number of layers. Also, recent CNN architectures like resnet pass residual info from layers to layers, eliminating in practice vanishing gradient. For exploding gradient, you see if if loss suddenly explodes.", "529351": "vivaroma, a small gradient means that the weights and biases of the initial layers will not be updated effectively with each training session. Since these initial layers are often crucial to recognizing the core elements of the input data, it can lead to overall inaccuracy of the whole network.", "529443": "I see...Do you actively check the gradients in practice to avoid vanishing gradient? Sounds like it only happens under certain conditions/architectures, and we might not need to worry about it otherwise.", "529666": "As everyone else said. I'll also add that vanishing gradient affected specially Vanilla RNN because propagating the error along the time made the gradient to become exponentially small for long time sequences. LSTM and GRU solves this issue teorically but on practice you have to be carefull because with some really long sequences you will most probably have vanishing gradient.\n\nIf you are interested more on this topic I'll suggest reading this [post](https://medium.com/datadriveninvestor/how-do-lstm-networks-solve-the-problem-of-vanishing-gradients-a6784971a577) which explains everything really well.", "531785": "How do you deal with exploding gradient then?", "531798": "lower learning rate, clip gradient, check sample weights, ..." }, "source": "meta" }