{"cells":[{"metadata":{},"cell_type":"markdown","source":"### In probability theory and statistics, the Laplace distribution is a continuous probability distribution named after Pierre-Simon Laplace. It is also sometimes called the double exponential distribution, because it can be thought of as two exponential distributions spliced together back-to-back.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://upload.wikimedia.org/wikipedia/commons/thumb/0/0a/Laplace_pdf_mod.svg/400px-Laplace_pdf_mod.svg.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"[Image Credits: Wikipedia](https://en.wikipedia.org/wiki/Laplace_distribution)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Probability density function\n\n![](https://wikimedia.org/api/rest_v1/media/math/render/svg/210306e18c75c252ce85eb79c3af18bb5c8dd1a8)\n![](https://wikimedia.org/api/rest_v1/media/math/render/svg/c7092c31ac10642b5e709ef827893bc2e18ec580)\n\nHere, mu  is a location parameter and b > 0, which is sometimes referred to as the diversity, is a scale parameter. If mu =0 and b=1, the positive half-line is exactly an exponential distribution scaled by 1/2.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> The probability density function of the Laplace distribution is also reminiscent of the normal distribution; however, whereas the normal distribution is expressed in terms of the squared difference from the mean mu , the Laplace density is expressed in terms of the absolute difference from the mean. **Consequently, the Laplace distribution has fatter tails than the normal distribution.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### **After Reading articles online for a while I found that Laplace Distribution is quite popular metric for Deep Learning and there is a reason behind this**.\n#### I found the following answer on [Why deep learning prefer the probability distribution with a sharp point?](https://stats.stackexchange.com/questions/275779/why-deep-learning-prefer-the-probability-distribution-with-a-sharp-point) quite interesting","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### In Ian Goodfellow's book about deep learning when it introduces exponential distribution, it says \"In the context of deep learning, we often want to have a probability distribution with a sharp point at x=0.\" \n\n![](https://i.stack.imgur.com/tCTsY.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### But the question is, why the probability distribution should have this character, does it help to train the model?\n\n#### The above 2(exponential and laplace) distributions have a connection with deep learning via regularisation. In deep learning we are often concerned with regularising the parameters of a neural network because neural networks tend to overfit and we want to improve ability of the model to generalise to new data.\n\n#### From a Bayesian perspective, fitting a regularised model can be interpreted as computing the maximum a posteriori (MAP) estimate given a specific prior distribution over the weights wi. In particular,the L2 (a.k.a. weight decay) norm corresponds to a Gaussian prior on the weights w, and the L1 norm corresponds to an isotropic Laplace prior over the weights w.\n\n#### The L1 norm (a.k.a. the Laplace prior on weights) by virtue of it's sharpness encourages sparsity (many zeros) in w.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}