{"cells":[{"metadata":{"_uuid":"34dd29a880fe4ca5c3f2e51c026db7bdcbcfcb7b"},"cell_type":"markdown","source":"# **BIG DISCLAIMER READ THIS FIRST !**\n\n### Hi ! Welcome to my newest work in progress ! Thanks for passing by !! \nFew things to begin with : this is me sharing one of my biggest project to date, building a fully functionning neural network with countless features from scratch, in C++. I want to tell you about the philosphy behind this so, I've recently been adapting my code in Python at an attempt to put it on GitHub for the world to see.\n\nI'll uptdate this kernel **DAILY** as this is now my primary project on deep learning. \n\n**PLEASE PLEASE PLEASE PRETTY PLEASE !** tell me what you think about it, do you like the idea ? do you not like it ? do you think it will be useful ? will you use it ? how well is it presented ? what would you add ? etc. etc. If you give feedback I'll make sure to mention you on this kernel (in a section that will be at the top of this), in my future github repo with the link towards ***YOUR*** GitHub and if this goes somewhere else, I'll mention you there too !!\n\n### **Second big disclaimer**\n\nI'm French, so my code was in French, the classes were in French and the comments were in French, therefore, you might still find some French here and there, I'll make sure to **eradicate** it as fast as I can, but first, I need to make sure no conflict is made with the other classes on the full code (which is not yet available here, as I will update on the days to come). If you see some French, don't feak out, it will change for clarity. "},{"metadata":{"_uuid":"f1538dacb872eda9784a61fef986ab437fa47a83"},"cell_type":"markdown","source":"#### **Versions Journal**\n\n* 31/01/2019 : First commit, the backbone is there and the philosphy of the project is explained."},{"metadata":{"_uuid":"3a5eb3808e9994be3a372c08c2ac2d4588d5bdb8"},"cell_type":"markdown","source":"## **Hey ! ** Thanks for clicking this link, while you are here, let me explain you what this is all about. \n\n> Section in progress [##########---] 80% completed, it misses some cool images\n\n#### Sooo... If you clicked here, there is a good chance you are \"familiar\" with Deep Learning, and actually, if you have done few projects you know what the deep learning pipeline looks like : \n\n1. Look at data\n2. Extract features which look promising (based on statistics)\n3. Engineer those features\n4. Build a deep learning model\n5. Train it (and verify it with cross validation)\n6. Test it and submit  your answers\n7. ?? .. Profit.\n\nBut you know more than me, when you do all these things, it doesn't always (actually it almost never) does what you want and you end up with results which have poor accuracy or they *overfit* or god knows what else. \n\nWhile making sure your data has high quality standard, using **statistic tests** and that your features are not **disproportionate** and not **correlated** (and while we are at it, to make sure that your train set and your test set are also not correlated) might solve some of your problems, sometimes it just doesn't make it.\n\nWe can spend hours and hours building a deep learning network to get 60% accuracy on the test set (which is slightly better than random for a 2-class classification), and when you think about it, that's kind of depressing.\n\n The answers lies in the 4. of the deep learning pipeline, and about that I would like to ask a simple question : How much do you really understand your deep learning model?\n \n There is a good chance you just build it with TensorFlow or Keras and you just pile on these hidden layers, drop out layers, with LeakyRelu because somewhere you read that it gave \"the best performance\", and you give it something, and it just returns error after error and eventually, with perseverence (and StackOverflow) you make it work.\n \n But after all, how good are you at explaining what **exactly** is inside the network and why is it there ? why LeakyRelu and not just Relu ? Why RMSE ? why are your weights initialized this way ? why your dropout is 25% and not 27.7% ? why that convolution kernel is (3,3) and not (3,4) ? Often the answers to these questions are : because it gives better results this way. \n Deep Learning then becomes a contest about who can run a GridSearchCV like algorithm on the most parameters to find the best set of parameters. That's not very exciting in my humble opinion, so let me ask this next question : What if we could get it right from the **first time.**\n \n So, in order to build and **understand** your network, to see it evolve in front of your very eyes, and to make decisions according to the nature of the data and **not** based on an accuracy score, I would like to take you on the journey of deep learning, **without** Keras or Tensorflow or anything else, just good ole' Python, Numpy and Math."},{"metadata":{"_uuid":"e2422e47a4d05a8fc0c6641c22d363b53f9c42c5"},"cell_type":"markdown","source":"## Why ditch (temporarly) the deep learning frameworks and build your own model in a true DIY fashion ?\n\n> Section in progress [##########---] 80% completed, it misses some cool images\n\n### Reason 1 : Knowledge\n\nThis was the most motivating reason, for me. To know and understand what is in that black box. By building it yourself, you will have to deeply (no pun intended) understand what is inside the neural nets, you will have to dive deep into the math and algebra that define the backpropagation algorithm, you will finally know exactly what's the difference between ReLu and sigmoid. That's the obvious knowledge, but there is more ! You'll learn object oriented programming, you will learn to optimize your code (altough one can argue that going at it with Python is not the smartest decision on that one) and also,  you'll conquer the blank paper syndrom.\n\n### Reason 2 : Full Control\n\nThe power will be in your hands from now on, you'll not depend on which release of TensorFlow is out there, you'll not depend on how their dropout algorithm works, you will be able to track each particular weight of your network and see how it affects the result, and as your advance and code more features (convolution, dropout, reccurent, Q-learning...)  you'll be able to build more and more complex models to feet your needs\n\n### Reason 3 : Sandbox mode\n\nPerhaps the most interesing reason why building your own models is good, is that you can do things which have not been done before. Your imagination is the only limit\n\n#### Should you forget Tensorflow forever ? \n\nEeeeeh probably not, let's keep it real here, a certain number of Google engineers work on this every day, not sure you can keep up...  plus TensorFlow is written in C and optimized for multithreading as well as GPU support, it can be deployed on mobile devices (TFlite), exported to the web and comes with another handful of cool features (TensorBoard anybody ?) and the same goes for Keras, for Pytorch and other framework out there. So you'll probably have to return to one of those popular framework but you will with a better understanding of the problem, you will also reads the doc with better attention, and why not go and see the code yourself ! Hopefully you will build your networks with more care, and instead of just plugging those parameters in, you'll handpick each parameter and you will be confident on your results."},{"metadata":{"_uuid":"2b0cc75afd46e94ffad6aa199b7855f36f78f68f"},"cell_type":"markdown","source":"## So, What do we need ? \n> Section in progress [#------------------------] 5% completed, I need to write a lot here, explain key concepts and find some cool images\n### 1. We need to know about neural networks\n\n### 2. We need to know about object oriented programming\n\n### 3. We need to have a clear idea on how this whole system will be organized and we need to think ahead of time for future improvements"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"565e8a42d84e930fc64b2174dc3529f54428e884"},"cell_type":"markdown","source":"> TODO : introduce sigmoid, ReLu and more friends"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"def sigmoid(x):\n        y =1/(1 + exp(-x))\n        return y\n    \n#Todo : ReLu and friends definition","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f8496bf20e9ca5391ca449b0cd8875385f7f7610"},"cell_type":"markdown","source":"> TODO : test sigmoid and ReLu on few values and give a conclusion (sigmoid is highly sensitive arround 0...)"},{"metadata":{"_uuid":"fd30b823f6e76c88e345218788fcb8416c7d8cd8"},"cell_type":"markdown","source":"> TODO : introduce cost functions (RMSE and friends)"},{"metadata":{"trusted":true,"_uuid":"02507e2a8b97403eb8e484cca5a4b7cdfb7b2f92"},"cell_type":"code","source":"def RMSE(x,y):\n        z = (x-y)**2\n        return z\n\n#Todo : friends definition","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"09d002c8964c318a61a8760275048613aafbd877"},"cell_type":"markdown","source":"> TODO : test cost functions on few values"},{"metadata":{"_uuid":"ae8c65e4879001b084c4c6b8bbb22692ef37ac5c"},"cell_type":"markdown","source":"> BIG SECTION TODO : Introduce the Perceptron class"},{"metadata":{"trusted":true,"_uuid":"fc6153ed45b1b540f08ba103c48c55b036f6dbd3"},"cell_type":"code","source":"class Perceptron():\n\n    #La classe perceptron correspond à la définition d'un perceptron d'un réseau neuronal profond.\n\n    def __init__(self,function,entry=3,bias=0.):\n        self.function = function #fonction de normalisation\n        self.entry = entry #taille du vecteur d'entrée\n        self.weights =np.linspace(0.,1.,entry) #initialisation des poids\n        self.bias = bias #initialisation du biais\n\n    def set_function(self,function):\n        #met à jour la fonction de normalisation\n        self.function = function\n\n    def set_weights(self,weights):\n        #met à jour les poids\n        self.weights=weights\n\n    def set_bias(self,biais):\n        #met à jour le biais\n        self.biais = biais\n\n\n    def compute(self,input):\n    #Compute calcul la sortie d'un perceptron.\n        output = 0\n        #Erreur si la dimension du vecteur en entrée ne correspond pas\n        if len(input) != len(self.weights):\n            raise NameError('La dimension du vecteur en entrée ne correspond pas avec la dimension prédéfinie pour le perceptron.')\n\n        #Calcul FeedForward\n        for i in range(len(input)):\n            output = output+self.weights[i]*input[i]\n        output = output+self.bias\n        output=self.function(output)\n        return output","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f0b2496d08eef209e2e0017bfc80fde0cbabfd3a"},"cell_type":"markdown","source":"> TODO : test the Perceptron class"},{"metadata":{"_uuid":"f5adde6524e358fe2f534c6ecfdec27cc279c029"},"cell_type":"markdown","source":"> BIG TODO : define the layer class"},{"metadata":{"trusted":true,"_uuid":"91b787e8f84b2e843d3537c0455ac3c158697517"},"cell_type":"code","source":"class Couche():\n\n    #La classe couche correspond à la définition d'une couche composée de plusieurs neurones.\n\n    def __init__(self,nb_neurones,function,entry):\n        #initialise les perceptrons de la couche\n        neurones = [Perceptron(function,entry)]\n        for i in range(nb_neurones-1):\n            neurones.append(Perceptron(function,entry))\n        self.neurones = neurones\n        self.nb_neurones = nb_neurones\n\n    def compute_couche(self,input):\n        #calcul la sortie d'une couche\n        result = np.zeros(self.nb_neurones)\n        for i in range(self.nb_neurones):\n            result[i] = self.neurones[i].compute(input)\n        return result","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"07f05c63af16e37e36cc3a3d4f1aea76cf6fff1e"},"cell_type":"markdown","source":"> TODO : test the layer class"},{"metadata":{"_uuid":"5fb602ff05dedb1515df9f25217704134459eed1"},"cell_type":"markdown","source":"> Big TODO : define the network class"},{"metadata":{"trusted":true,"_uuid":"631a143a0d8514377968210620ceca8d89fe6292"},"cell_type":"code","source":"class Reseau():\n\n    #La classe Reseau correspond à l'ensemble du réseau de neurones \"fully connected\"\n\n    def __init__(self,input_size,function,cost_function,nb_couches=3\n            ,couche_size=3,output_size=2):\n\n        self.input_size = input_size\n\n        #initialise les couches du réseau\n        couches = [Couche(couche_size,function,entry = self.input_size)]\n        for i in range(nb_couches-1):\n            couches.append(Couche(couche_size,function,entry=couche_size))\n\n        #initialisation de la fonction coût\n        self.cost_function = cost_function\n\n        #ajout de la couche de sortie\n        couches.append(Couche(output_size,function,entry=couche_size))\n\n        #fin de l'initialisation des couches du réseau\n        self.couches = couches\n\n    def feed_forward(self,entree):\n\n        #Cette fonction calcule la sortie du réseau de neurones\n\n        if len(entree) != self.input_size:\n            raise NameError(\"l'entrée n'est pas de la bonne taille\")\n        output = np.copy(entree)\n        for i in range(len(self.couches)):\n            output = self.couches[i].compute_couche(output)\n\n        return output\n    \n    #todo : define the training function with the backprop. algorithm.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"155e55602cbd4ba969fdadd2f87d504a4b6d7f45"},"cell_type":"markdown","source":"> TODO : test the network class"},{"metadata":{"trusted":true,"_uuid":"f9a3b2feb7d6a701dd32789c720d1aeadcdd3c7a"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}