{"cells":[{"metadata":{"_uuid":"80674ffcc32a69c06837ab064e12b69665132571"},"cell_type":"markdown","source":"

\n \n \n \n

\n\n---\n\n# Tensorflow Tutorial\n---\n> * **Tensorflow Introduction**\n * [**Introduction of Tensorflow **](#Introduction-of-Tensorflow )\n * [**TensorFlow Environment Configuration**](#TensorFlow-Environment-Configuration)\n* **Section - 1 Tensorflow basic**\n * [**Variable**](#1.Variable)\n * [**Constant**](#2.Constant)\n * [**Placeholder**](#3.Placeholder)\n * [**Session**](#4.Session)\n * [**Graph**](#5.Graph)\n * [**Activation**](#6.Activation)\n* **Section - 2 Build your first network (Neural Network)**\n * [**Regression**](#Regression)\n * [**Classification**](#Classification)\n * [**Save and reload**](#Save-and-reload)\n * [**Optimizers**](#Optimizers)\n * [**Tensorboard**](#Tensorboard)\n * [**Dataset**](#Dataset)\n* **Section - 3 Advanced neural network ( Deep Learning)**\n * **Supervise Deep learning**\n * [**CNN**](#CNN)\n * [**RNN Classification**](#RNN-Classification)\n * [**RNN-Regression**](#RNN-Regression)\n * **Unsupervise Deep Learning**\n * [**AutoEncoder**](#AutoEncoder)\n * [**GAN (Generative Adversarial Nets)**](#GAN-(Generative-Adversarial-Nets)) / [**Conditional GAN**](#Conditional-GAN)\n * [**Transfer Learning**](#Transfer Learning)\n * **Reinforcement Learning**\n * [**DQN Reinforcement Learning**](#DQN-Reinforcement-Learning)\n* **Section - 4 Others (WIP)**\n * [**Dropout**](#Dropout)\n * [**Batch Normalization**](#Batch-Normalization)\n * [**Visualize Gradient Descent**](#Visualize-Gradient-Descen)\n * [**Distributed training**](#Distributed-training)\n \n \n ---"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"\n### Introduction of Tensorflow\n\n![](https://storage.googleapis.com/kaggle-datasets/94840/220637/tensorflow.jpg?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1545649759&Signature=oqKqxAogetwl%2BR8gg35UOMlii7Vrg7lsQmarC1sa0%2FTrwYMOnt373lUSjM5HeU9JrchiwKkn%2FT2Kngtbscufk5PUJiqsV41aSwrJ%2FEwo1XZJdTmbDmUD%2FSAMVvkcJ8PMd%2FtX3vw7hQGojmRujw%2BLNKEtIdkuzDf7nWfIzgVmbC3afpxoLSO%2BpFQHdpfcogZv9cY8ntPTF5cfKKxtgEveNq2Fx245upShjgXOoWQxKzPHQKmPih3GYGPRbuEHChq2ijrO%2BL4O9aD1WIV6JNpEMN8hA00NUe%2B2mpW5zM%2BKprWKoWx46aaxb8qiT1ldusyK4ANHScdcpjEu5uCSUdNSig%3D%3D) \n\n* **TensorFlow** is an interface for expressing **machine learning algorithms**, and an implementation for executing such algorithms. A **computation expressed using TensorFlow** can be executed with little or no change on a **wide variety of heterogeneous systems**, ranging from **mobile devices such as phones and tablets** up to **large-scale distributed systems** of hundreds of machines and thousands of computational devices such as **GPU cards.** The system is flexible and can be used to express a wide variety of algorithms, including **training and inference algorithms for deep neural network models**, and it has been used for conducting research and for deploying **machine learning systems into production across more than a dozen areas of computer science** and other fields, including **speech recognition, Computer vision, robotics, information retrieval, natural language processing, geographic information extraction, and computational drug discovery.**\n\n**References : [Tensorflow White Paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45166.pdf)**"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"### TensorFlow Environment Configuration\n\n![](https://www.kaggle.com/ashishpatel26/tensorflow-tutorial-data/downloads/install.JPG)\n\n**Resources to help tensorflow :**\n* https://www.tensorflow.org/install/pip - Official Tutorial\n* https://saintlad.com/install-tensorflow-on-windows/"},{"metadata":{"trusted":true,"_uuid":"bf53496725c9b740f4e7727a60cb61b871845449"},"cell_type":"markdown","source":"# Section - 1 Tensorflow basic\n---\n### Let's We go through Basic\n\n### 1. What is Tensor?\n\n---\n\n![](https://www.kdnuggets.com/wp-content/uploads/scalar-vector-matrix-tensor.jpg)\n\n* **Tensorflow's** name is directly derived from its **core framework: Tensor.** In **Tensorflow,** all the computations involve **tensors.** \n* **A tensor** is a **vector or matrix of n-dimensions that represents all types of data.** All values in a **tensor hold identical data** type with a **known (or partially known) shape.** \n* The **shape of the data** is the **dimensionality of the matrix or array.**\n\n### 2.Representation of a Tensor\n\n---\n* In **TensorFlow, a tensor is a collection of feature vectors (i.e., array) of n-dimensions.**\n* For instance, if we have a **2x3 matrix** with values from 1 to 6, we write:\n\n![](https://www.guru99.com/images/1/080418_1250_WhatisaTens111.png)\n\n![](https://www.guru99.com/images/1/080418_1250_WhatisaTens2.png)\n\n### 3.Types of Tensor\n\n---\n![](http://hpe-cct.github.io/programmingGuide/img/diagram1.png)\n\n#### In **TensorFlow**, all the computations pass through one or more tensors. A tensor is an object with **three properties:**\n\n* **A unique label (name)**\n* **A dimension (shape)**\n* **A data type (dtype)**\n\n### 4. Operation that doing in Tensorflow\n\n---\n#### **TensorFlow** involves the **manipulation of a tensor.** There are four main tensors you can create:\n\n* **tf.Variable** \n* **tf.constant**\n* **tf.placeholder**\n* **tf.SparseTensor**\n\n![](https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/image_folder_7/Tensorflow_Graph_0.png)\n![](https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/image_folder_7/tensors_flowing.gif)"},{"metadata":{"_uuid":"903f27b3a77b0193d9fa6fffac7a4dc2410497a6"},"cell_type":"markdown","source":"## 1.Variable\n[**Go to Top**](#Tensorflow-Tutorial)\n\n### tf.varible\nClass **Variable**\n\n---\n\n* A TensorFlow **variable** is the best way to represent shared, persistent state manipulated by your program.\n* Variables are manipulated via the [**`tf.Variable`**](https://www.tensorflow.org/api_docs/python/tf/Variable) class. A [**`tf.Variable`**](https://www.tensorflow.org/api_docs/python/tf/Variable) represents a tensor whose value can be changed by running ops on it. Unlike[**`tf.Variable`**][**`tf.Variable`**](https://www.tensorflow.org/api_docs/python/tf/Tensor) objects, a[**`tf.Variable`**](https://www.tensorflow.org/api_docs/python/tf/Variable) exists outside the context of a single `session.run` call.\n* Internally, a [**`tf.Variable`**](https://www.tensorflow.org/api_docs/python/tf/Variable) stores a persistent tensor. Specific ops allow you to read and modify the values of this tensor. These modifications are visible across multiple [**`tf.Session`**](https://www.tensorflow.org/api_docs/python/tf/Session)s, so multiple workers can see the same values for a [**`tf.Variable`**](https://www.tensorflow.org/api_docs/python/tf/Variable).\n\n\n\n"},{"metadata":{"trusted":true,"_uuid":"934817c52b464ce0182e9a11100c53572a38d5b3"},"cell_type":"code","source":"import tensorflow as tf\n\nvar = tf.Variable(0) # our first variable in the \"global_variable\" set","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"df5b4dda00e5fe62eb3f6775dab416cb7b57a30a"},"cell_type":"code","source":"add_operation = tf.add(var, 1)\nupdate_operation = tf.assign(var, add_operation)\n\nwith tf.Session() as sess:\n # once define variables, you have to initialize them by doing this\n sess.run(tf.global_variables_initializer())\n for _ in range(3):\n sess.run(update_operation)\n print(sess.run(var))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e80fcedf7212511b266853109f250d9b7064a900"},"cell_type":"markdown","source":"## 2.Constant\n[**Go to Top**](#Tensorflow-Tutorial)\n\n### tf.constant\n\n---\n\n```python\ntf.constant( \n    value, \n    dtype=None, \n    shape=None, \n    name='Const', \n    verify_shape=False \n) \n```\n* ***Creates a constant tensor.***\n* The resulting tensor is **populated with values of type `dtype`, as specified by arguments `value`** and **(optionally) `shape`.**\n* The argument **`value`** can be a **constant value**, or a **list of values of type `dtype`.** If **`value`** is a **list**, then **the length of the list must be less than or equal to the number of elements implied by the `shape` argument** (if specified). In the case where the list length is less than the number of elements specified by `shape`, the last element in the list will be used to fill the remaining entries.\n* The argument **`shape`** is optional. If present, it ***specifies the dimensions of the resulting tensor.** If **not present, the shape of `value` is used.**\n* If the argument **`dtype`** is **not specified,** then the **type is inferred from the type of `value`.**"},{"metadata":{"trusted":true,"_uuid":"0b9ac30404b4eac0222929f790b29ef837b93152"},"cell_type":"code","source":"# Constant 1-D Tensor populated with value list.\ntensor_1 = tf.constant([1, 2, 3, 4, 5, 6, 7])\nprint(tensor_1.shape)\n# Constant 2-D tensor populated with scalar value -1.\ntensor_2 = tf.constant(-1.0, shape=[2, 3])\nprint(tensor_2.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5146de06f721767a9097fed59115274da25b966c"},"cell_type":"markdown","source":"## 3.Placeholder\n[**Go to Top**](#Tensorflow-Tutorial)\n\n### tf.placeholder\n\n---\n```python\ntf.placeholder(\n dtype,\n shape=None,\n name=None\n) ```\n\n* A **placeholder** has the **purpose of feeding the tensor.** Placeholder is used to **initialize the data to flow inside the tensors**. To **supply a placeholder**, you need to use the **method feed_dict**. The placeholder will be fed only within a session.\n* In the next example, you will see how to create a placeholder with the method tf.placeholder. In the next session, you will learn to fed a placeholder with actual value."},{"metadata":{"trusted":true,"_uuid":"aa1f6eb1a2fefc86b709be4e3122b3a163c726b4"},"cell_type":"code","source":"import tensorflow as tf\n\ntf.reset_default_graph() # To clear the default graph \n\nx1 = tf.placeholder(dtype=tf.float32, shape=None) # Inserts a placeholder for a tensor that will be always fed.\ny1 = tf.placeholder(dtype=tf.float32, shape=None) # Inserts a placeholder for a tensor that will be always fed.\nz1 = x1 + y1 # Summation `x1` and `y1`","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"995d4cc8390e9fa9b36e3b87184a0b13f2228fd5"},"cell_type":"code","source":"x2 = tf.placeholder(dtype=tf.float32, shape=[2, 1]) # Inserts a placeholder for a tensor that will be always fed.\ny2 = tf.placeholder(dtype=tf.float32, shape=[1, 2]) # Inserts a placeholder for a tensor that will be always fed.\nz2 = tf.matmul(x2, y2) # Multiplies matrix `x2` by matrix `y2`, producing `x2` * `y2`","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dd595e33f85d6b5c32ff672cbbce6f48ed711465"},"cell_type":"code","source":"with tf.Session() as sess:\n # when only one operation to run\n z1_value = sess.run(z1, feed_dict={x1: 1, y1: 2}) # Runs operations and evaluates tensors in `fetches`.\n\n # when run multiple operations\n z1_value, z2_value = sess.run(\n [z1, z2], # run them together\n feed_dict={ # Define all variable value in placeholder\n x1: 1, y1: 2,\n x2: [[2], [2]], y2: [[3, 3]]\n })\n print(z1_value) # print value of z1\n print(z2_value) # print value of z2","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"573d621a757d4e2d912a52fb3cddc79261a9ce07"},"cell_type":"markdown","source":"## 4.Session \n\n[**Go to Top**](#Tensorflow-Tutorial)\n\n### tf.session\nClass **Session**\n\n---\n\n* A class for running **TensorFlow operations.**\n* A **Session object encapsulates the environment in which Operation objects are executed, and Tensor objects are evaluated.** \n\n#### TensorFlow works around 3 main components:\n\n| **Components**| **Descritption**|\n| --- | --- |\n| **Graph** | The graph is fundamental in TensorFlow. All of the mathematical operations (ops) are performed inside a graph. You can imagine a graph as a project where every operations are done. The nodes represent these ops, they can absorb or create new tensors. |\n| **Tensor** | A tensor represents the data that progress between operations. You saw previously how to initialize a tensor. The difference between a constant and variable is the initial values of a variable will change over time. | \n| **Session** | A session will execute the operation from the graph. To feed the graph with the values of a tensor, you need to open a session. Inside a session, you must run an operator to create an output. |\n\n\n**Here two different methods explained.**"},{"metadata":{"trusted":true,"_uuid":"6b0bf5dd6855d62a6a8f29ea671c16ee35997060"},"cell_type":"code","source":"import tensorflow as tf\n\nm1 = tf.constant([[2, 2]])\nm2 = tf.constant([[3],\n [3]])\ndot_operation = tf.matmul(m1, m2)\n\nprint(dot_operation) # wrong! no result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bb587dda281d966550dc98769108a65b2937951d"},"cell_type":"code","source":"# method1 use session\nsess = tf.Session()\nresult = sess.run(dot_operation)\nprint(result)\nsess.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0feb55d7ec7cace11e2044f12d7302ce05769999"},"cell_type":"code","source":"# method2 use session\nwith tf.Session() as sess:\n result_ = sess.run(dot_operation)\n print(result_)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0d35edbd46eb9cc154aec80c677f4648137fac50"},"cell_type":"markdown","source":"## 5.Graph\n[**Go to Top**](#Tensorflow-Tutorial)\n\n---\n\n* **TensorFlow depends on a genius approach to render the operation.** All the *computations are represented with a dataflow scheme.* The *dataflow graph* has been developed to see to *data dependencies between individual operation. Mathematical formula or algorithm are made of a number of successive operations. A graph is a convenient way to visualize how the computations are coordinated.*\n\n* *The graph shows **a node and an edge.** The **node is the representation of a operation**, i.e. **the unit of computation.** The **edge** is the **tensor, it can produce a new tensor or consume the input data.** It depends on the dependencies between individual operation.*\n\n* *The structure of the **graph connects together the operations (i.e. the nodes)** and **how those are operation are feed.** Note that the **graph does not display the output of the operations**, it **only helps to visualize the connection between individual operations.***\n\n#### Let's see an example.\n\n**Imagine you want to evaluate the following function:**\n\n![](https://www.guru99.com/images/1/080418_1250_WhatisaTens41.jpg)\n\n**TensorFlow will create a graph to execute the function. The graph looks like this:**\n\n![](https://www.guru99.com/images/1/080418_1250_WhatisaTens4.png)"},{"metadata":{"trusted":true,"_uuid":"ba8e354a0b6c3e1adc77d8d4465566e493fe6e29"},"cell_type":"code","source":"tf.reset_default_graph() # To clear the default graph \nx = tf.get_variable(\"x\", dtype=tf.int32, initializer=tf.constant([5]))\nz = tf.get_variable(\"z\", dtype=tf.int32, initializer=tf.constant([6]))\nc = tf.constant([5], name =\t\"constant\")\nsquare = tf.constant([2], name =\"square\")\nf = tf.multiply(x, z) + tf.pow(x, square) + z + c","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1f1de919792c75417a6081e9fa075ac588b657d5"},"cell_type":"code","source":"init = tf.global_variables_initializer() # prepare to initialize all variables\nwith tf.Session() as sess: \n init.run() # Initialize x and y \n function_result = f.eval()\nprint(function_result) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"11353582a428b74080d7d862826dcd8a7451784d"},"cell_type":"markdown","source":"## 6.Activation\n[**Go to Top**](#Tensorflow-Tutorial)\n\n![](https://cdn-images-1.medium.com/max/1600/1*RD0lIYqB5L2LrI2VTIZqGw.png)\n"},{"metadata":{"trusted":true,"_uuid":"5a010f9414c3208029df1f267dc6ca4fe6064f63"},"cell_type":"code","source":"import tensorflow as tf\nimport numpy as np\nimport matplotlib.pyplot as plt\nplt.style.use(\"seaborn-darkgrid\")\ntf.reset_default_graph() # To clear the default graph ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54a95b915789b0f2b724cc5d2499081912c0768b"},"cell_type":"code","source":"# fake data\nx = np.linspace(-5, 5, 200) # x data, shape=(100, 1)\n\n# following are popular activation functions\ny_relu = tf.nn.relu(x)\ny_sigmoid = tf.nn.sigmoid(x)\ny_tanh = tf.nn.tanh(x)\ny_softplus = tf.nn.softplus(x)\n# y_softmax = tf.nn.softmax(x) softmax is a special kind of activation function, it is about probability","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"abee4a3522322885c46d563de288739f1efe3016"},"cell_type":"code","source":"sess = tf.Session()\ny_relu, y_sigmoid, y_tanh, y_softplus = sess.run([y_relu, y_sigmoid, y_tanh, y_softplus])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2657742d4acd71e229aef7fa03d03185072f97d7"},"cell_type":"code","source":"# plt to visualize these activation function\nplt.figure(1, figsize=(20, 12))\nplt.subplot(221)\nplt.plot(x, y_relu, c='red', label='relu')\nplt.ylim((-1, 5))\nplt.legend(loc='best')\n\nplt.subplot(222)\nplt.plot(x, y_sigmoid, c='red', label='sigmoid')\nplt.ylim((-0.2, 1.2))\nplt.legend(loc='best')\n\nplt.subplot(223)\nplt.plot(x, y_tanh, c='red', label='tanh')\nplt.ylim((-1.2, 1.2))\nplt.legend(loc='best')\n\nplt.subplot(224)\nplt.plot(x, y_softplus, c='red', label='softplus')\nplt.ylim((-0.2, 6))\nplt.legend(loc='best')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"760a1ab9aebaecdb15a2758ddbf4d7328f0bf97c"},"cell_type":"markdown","source":"# Section - 2 Build your first network (Neural Network)\n---\n[**Go to Top**](#Tensorflow-Tutorial)\n\n### Regression\n---\n\nImage Reference : https://github.com/Avik-Jain/100-Days-Of-ML-Code\n\n![](https://github.com/Avik-Jain/100-Days-Of-ML-Code/raw/master/Info-graphs/Day%202.jpg)"},{"metadata":{"_uuid":"67827b53560f4c9790420722f772a6807f9e7a91"},"cell_type":"markdown","source":"### 1. Library"},{"metadata":{"trusted":true,"_uuid":"a4651cb3b9c5ba26ad076dc925c34fa449495061"},"cell_type":"code","source":"import tensorflow as tf\nimport matplotlib.pyplot as plt\nimport numpy as np\ntf.reset_default_graph() # To clear the default graph \n\ntf.set_random_seed(1)\nnp.random.seed(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e4d3a26f2f7e638ace3b520db359411b62f21eb"},"cell_type":"markdown","source":"### 2.Data generate"},{"metadata":{"trusted":true,"_uuid":"7effcf5e761f17734f0c5a0b35ac7b538510699a"},"cell_type":"code","source":"# fake data\nx = np.linspace(-1, 1, 100)[:, np.newaxis] # shape (100, 1)\nnoise = np.random.normal(0, 0.1, size=x.shape)\ny = np.power(x, 2) + noise # shape (100, 1) + some noise","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4734d0d41ed21a63657c236ee9f6956c61bf70fa"},"cell_type":"markdown","source":"### 3.plot data"},{"metadata":{"trusted":true,"_uuid":"4b52b7b2af2d62a30d78f550dffbec045c8da991"},"cell_type":"code","source":"plt.figure(figsize=(20,8))\nplt.scatter(x, y)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd7272b6f4c225a3f41366231147bae1a0afca2b"},"cell_type":"markdown","source":"### 4.Define data in place holder"},{"metadata":{"trusted":true,"_uuid":"9abc424de00c9eef5d78e829d586d50d598f00ea"},"cell_type":"code","source":"tf_x = tf.placeholder(tf.float32, x.shape) # input x\ntf_y = tf.placeholder(tf.float32, y.shape) # input y","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cfdbacce09c7c2e278871a6f7e432e2826f249c2"},"cell_type":"markdown","source":"### 5.Define Model"},{"metadata":{"trusted":true,"_uuid":"6f59e5e1a408a0a1c0587e3e3ee9d459d77722cd"},"cell_type":"code","source":"# neural network layers\nl1 = tf.layers.dense(tf_x, 10, tf.nn.relu) # hidden layer\noutput = tf.layers.dense(l1, 1) # output layer\n\nloss = tf.losses.mean_squared_error(tf_y, output) # compute cost\noptimizer = tf.train.GradientDescentOptimizer(learning_rate=0.5)\ntrain_op = optimizer.minimize(loss)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1224d73077c4f320c509452ae58212f1682373b6"},"cell_type":"markdown","source":"### 6.Session Start for Model training\n\n\n![](https://camo.githubusercontent.com/7491264fba17ff7eb3ec5cce2e0f8db3e58e1c7b/68747470733a2f2f6d6f7276616e7a686f752e6769746875622e696f2f7374617469632f726573756c74732f746f7263682f312d312d322e676966)"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"8aeb40326a7b348b26a5ddbc4993212c814ab4e7"},"cell_type":"code","source":"sess = tf.Session() # control training and others\nsess.run(tf.global_variables_initializer()) # initialize var in graph\n\nplt.ion() # something about plotting\n\nfor step in range(100):\n # train and net output\n _, l, pred = sess.run([train_op, loss, output], {tf_x: x, tf_y: y})\n if step % 5 == 0:\n # plot and show learning process\n plt.figure(figsize=(20,6))\n plt.cla()\n plt.scatter(x, y)\n plt.plot(x, pred, 'r-', lw=5)\n plt.text(0.5, 0, 'Loss=%.4f' % l, fontdict={'size': 20, 'color': 'red'})\n plt.pause(0.1)\n\nplt.ioff()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f918c99f5ebb56592fdf5ae0c7202cd9e3b1a4f7"},"cell_type":"markdown","source":"### Other Related Notebook for learning\n- **Linear Regression** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/linear_regression.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/linear_regression.py)). Implement a Linear Regression with TensorFlow.\n- **Linear Regression (eager api)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/linear_regression_eager_api.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/linear_regression_eager_api.py)). Implement a Linear Regression using TensorFlow's Eager API.\n- **Logistic Regression** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/logistic_regression.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/logistic_regression.py)). Implement a Logistic Regression with TensorFlow.\n- **Logistic Regression (eager api)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/logistic_regression_eager_api.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/logistic_regression_eager_api.py)). Implement a Logistic Regression using TensorFlow's Eager API.\n- **Nearest Neighbor** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/nearest_neighbor.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/nearest_neighbor.py)). Implement Nearest Neighbor algorithm with TensorFlow.\n- **K-Means** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/kmeans.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/kmeans.py)). Build a K-Means classifier with TensorFlow.\n- **Random Forest** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/random_forest.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/random_forest.py)). Build a Random Forest classifier with TensorFlow.\n- **Gradient Boosted Decision Tree (GBDT)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/gradient_boosted_decision_tree.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/gradient_boosted_decision_tree.py)). Build a Gradient Boosted Decision Tree (GBDT) with TensorFlow.\n- **Word2Vec (Word Embedding)** ([notebook](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/word2vec.ipynb)) ([code](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/2_BasicModels/word2vec.py)). Build a Word Embedding Model (Word2Vec) from Wikipedia data, with TensorFlow."},{"metadata":{"_uuid":"3e3c109a35806050dbe728d2d8100ac55a809858"},"cell_type":"markdown","source":"## Classification\n---\n[**Go to Top**](#Tensorflow-Tutorial)\n\n* **Classification** is a **supervised learning algorithm** that **models or predicts discrete random variables.** Classification is usually based on regression method extensions and is suitable for predicting a category (or probability of a category) rather than a continuous value.\n\nCommon classification methods include : \n1. **Logistic Regression:** Corresponds to the linear regression method, but uses the Sigmoid function to map the prediction to a value between 0 and 1.\n1. **Classification tree:** Corresponding regression tree, also known as classification regression tree (CART), divides the data set into different branches to achieve hierarchical classification.\n1. **Deep learning:** using multi-layer neural network classification\n1. **Support Vector Machine (SVM):** Calculate the distance between support vectors based on kernel functions and find the boundary that maximizes its spacing from the sample\n1. **Naive Bayes:** A Classification Method Based on Bayes' Theorem and Characteristic Condition Independent Hypothesis\n"},{"metadata":{"_uuid":"a3b96fdd51bd6284b6fe79ba2c0e4ffc0bc3d8e0"},"cell_type":"markdown","source":"### 1. Library"},{"metadata":{"trusted":true,"_uuid":"259787716c6f4535d6e67488db44242b42e32b3b"},"cell_type":"code","source":"import tensorflow as tf\nimport matplotlib.pyplot as plt\nimport numpy as np\ntf.reset_default_graph() # To clear the default graph \ntf.set_random_seed(1)\nnp.random.seed(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"621719be14c74ef697e152b74aed3bcd444018be"},"cell_type":"markdown","source":"### 2.Data Generation"},{"metadata":{"trusted":true,"_uuid":"f0f56d827522a9bbbc85c9cdcb40ae1b7b7bfe42"},"cell_type":"code","source":"# fake data\nn_data = np.ones((100, 2))\nx0 = np.random.normal(2*n_data, 1) # class0 x shape=(100, 2)\ny0 = np.zeros(100) # class0 y shape=(100, )\nx1 = np.random.normal(-2*n_data, 1) # class1 x shape=(100, 2)\ny1 = np.ones(100) # class1 y shape=(100, )\nx = np.vstack((x0, x1)) # shape (200, 2) + some noise\ny = np.hstack((y0, y1)) # shape (200, )","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4aed2d92491d6951e30a1a3dff9d000027aea4d6"},"cell_type":"markdown","source":"### 3.Plot data"},{"metadata":{"trusted":true,"_uuid":"e8154d5dc8a467cc29b4fdfb9f40ef01d16dafd3"},"cell_type":"code","source":"# plot data\nplt.figure(figsize=(10,8))\nplt.scatter(x[:, 0], x[:, 1], c=y, s=100, lw=0, cmap='RdYlGn')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c14dcb334c288dbeb8024fa9f1c6fe8aff2d677d"},"cell_type":"markdown","source":"### 4. Load data into placeholder"},{"metadata":{"trusted":true,"_uuid":"dcd9b4832b1954f344e5ae57eda64031749d07ab"},"cell_type":"code","source":"tf_x = tf.placeholder(tf.float32, x.shape) # input x\ntf_y = tf.placeholder(tf.int32, y.shape) # input y","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7c505c7a0459e80f99170b2c90ab732a2a04be0d"},"cell_type":"markdown","source":"### 5. Model Traning"},{"metadata":{"trusted":true,"_uuid":"c5f0c1547471965d9cb4f9a68230b63464bdfab3"},"cell_type":"code","source":"# neural network layers\nl1 = tf.layers.dense(tf_x, 10, tf.nn.relu) # hidden layer\noutput = tf.layers.dense(l1, 2) # output layer\n\nloss = tf.losses.sparse_softmax_cross_entropy(labels=tf_y, logits=output) # compute cost\naccuracy = tf.metrics.accuracy( # return (acc, update_op), and create 2 local variables\n labels=tf.squeeze(tf_y), predictions=tf.argmax(output, axis=1),)[1]\noptimizer = tf.train.GradientDescentOptimizer(learning_rate=0.05)\ntrain_op = optimizer.minimize(loss)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b32f6dfcd67a0cccf904d76ce78a12e1946343d"},"cell_type":"markdown","source":"### 6. Model Result"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"4a416562d636172e9d77469c51becd70f2823bcb"},"cell_type":"code","source":"sess = tf.Session() # control training and others\ninit_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())\nsess.run(init_op) # initialize var in graph\n\nplt.ion() # something about plotting\nfor step in range(40):\n # train and net output\n _, acc, pred = sess.run([train_op, accuracy, output], {tf_x: x, tf_y: y})\n if step % 2 == 0:\n # plot and show learning process\n plt.figure(figsize = (20,6))\n plt.cla()\n plt.scatter(x[:, 0], x[:, 1], c=pred.argmax(1), s=100, lw=0, cmap='RdYlGn')\n plt.text(1.5, -4, 'Accuracy=%.2f' % acc, fontdict={'size': 20, 'color': 'red'})\n plt.pause(0.1)\n\nplt.ioff()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8746d2474a8733ea2fa1ec243957a2cdba46f03b"},"cell_type":"markdown","source":"## Save and reload\n---\n[**Go to Top**](#Tensorflow-Tutorial)"},{"metadata":{"_uuid":"a690fcd9284b8721790e7945fa91d0c2015cf2fc"},"cell_type":"markdown","source":"### 1. Load Library and Generate Data"},{"metadata":{"trusted":true,"_uuid":"b495bc7e02b157d24eaf35573c3ea6664c9148a5"},"cell_type":"code","source":"import tensorflow as tf\nimport matplotlib.pyplot as plt\nimport numpy as np\ntf.reset_default_graph() # To clear the default graph \ntf.set_random_seed(1)\nnp.random.seed(1)\n\n# fake data\nx = np.linspace(-1, 1, 100)[:, np.newaxis] # shape (100, 1)\nnoise = np.random.normal(0, 0.1, size=x.shape)\ny = np.power(x, 2) + noise # shape (100, 1) + some noise","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"28859c26011bd4d81063ee565d99539999811c97"},"cell_type":"markdown","source":"### 2.Model Save"},{"metadata":{"trusted":true,"_uuid":"668f683aeb7edaa834ae50538158aa4368ceae4f"},"cell_type":"code","source":"def save():\n print('This is save')\n # build neural network\n tf_x = tf.placeholder(tf.float32, x.shape) # input x\n tf_y = tf.placeholder(tf.float32, y.shape) # input y\n l = tf.layers.dense(tf_x, 10, tf.nn.relu) # hidden layer\n o = tf.layers.dense(l, 1) # output layer\n loss = tf.losses.mean_squared_error(tf_y, o) # compute cost\n train_op = tf.train.GradientDescentOptimizer(learning_rate=0.5).minimize(loss)\n\n sess = tf.Session()\n sess.run(tf.global_variables_initializer()) # initialize var in graph\n\n saver = tf.train.Saver() # define a saver for saving and restoring\n\n for step in range(100): # train\n sess.run(train_op, {tf_x: x, tf_y: y})\n\n saver.save(sess, './params', write_meta_graph=False) # meta_graph is not recommended\n\n # plotting\n pred, l = sess.run([o, loss], {tf_x: x, tf_y: y})\n plt.figure(1, figsize=(20, 5))\n plt.subplot(121)\n plt.scatter(x, y)\n plt.plot(x, pred, 'r-', lw=5)\n plt.text(-1, 1.2, 'Save Loss=%.4f' % l, fontdict={'size': 15, 'color': 'red'})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"70948187d27020550c58a8f3c4ac40a6d0d8a368"},"cell_type":"markdown","source":"### 3. Model Reload"},{"metadata":{"trusted":true,"_uuid":"728399c1b092c0d1768796ae2354b5d5e8629c09"},"cell_type":"code","source":"def reload():\n print('This is reload')\n # build entire net again and restore\n tf_x = tf.placeholder(tf.float32, x.shape) # input x\n tf_y = tf.placeholder(tf.float32, y.shape) # input y\n l_ = tf.layers.dense(tf_x, 10, tf.nn.relu) # hidden layer\n o_ = tf.layers.dense(l_, 1) # output layer\n loss_ = tf.losses.mean_squared_error(tf_y, o_) # compute cost\n\n sess = tf.Session()\n # don't need to initialize variables, just restoring trained variables\n saver = tf.train.Saver() # define a saver for saving and restoring\n saver.restore(sess, './params')\n\n # plotting\n pred, l = sess.run([o_, loss_], {tf_x: x, tf_y: y})\n plt.subplot(122)\n plt.scatter(x, y)\n plt.plot(x, pred, 'r-', lw=5)\n plt.text(-1, 1.2, 'Reload Loss=%.4f' % l, fontdict={'size': 15, 'color': 'red'})\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1dbbcf82b494288b6409c9e102b196e10f8f7032"},"cell_type":"markdown","source":"### 4.Model Save Loss and Reload Model Loss"},{"metadata":{"trusted":true,"_uuid":"d0c4ae71fcfa2f6d395dc80166b2d98e89006a5e"},"cell_type":"code","source":"save()\n\n# destroy previous net\ntf.reset_default_graph()\n\nreload()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"262038f4195ed4202ec4bef61a9574491fa696cc"},"cell_type":"markdown","source":"## Optimizers\n---\n\n[**Go to Top**](#Tensorflow-Tutorial)\n\n![](https://www.kaggle.com/ashishpatel26/dpandrewng/downloads/8.jpg)"},{"metadata":{"_uuid":"9df861db23974f09a62aa565bd87f50a20666a28"},"cell_type":"markdown","source":"### 1.Load Library"},{"metadata":{"trusted":true,"_uuid":"f7aa3bccd7a1d4c6ca6dec32d8a0ce65ec05aa39"},"cell_type":"code","source":"import tensorflow as tf\nimport matplotlib.pyplot as plt\nimport numpy as np\ntf.reset_default_graph() # To clear the default graph \ntf.set_random_seed(1)\nnp.random.seed(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b9a619f5822efefa5f5161a0ae4628f00b64211"},"cell_type":"markdown","source":"### 2.Define Parameter and Generate Data"},{"metadata":{"trusted":true,"_uuid":"421c6baa4691eef3f443f15493f916a9b92474a0"},"cell_type":"code","source":"LR = 0.01\nBATCH_SIZE = 32\n\n# fake data\nx = np.linspace(-1, 1, 100)[:, np.newaxis] # shape (100, 1)\nnoise = np.random.normal(0, 0.1, size=x.shape)\ny = np.power(x, 2) + noise # shape (100, 1) + some noise","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c049359752c3fb8324df30efe16c1d9a85c48ec4"},"cell_type":"markdown","source":"### 3.Plot Data"},{"metadata":{"trusted":true,"_uuid":"df1cf824f7ad3bd2b63bcdc42111a85ecfa41889"},"cell_type":"code","source":"# plot dataset\nplt.figure(figsize=(20,8))\nplt.scatter(x, y)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa42a10be88fbf61ba8ce0238f8b4e849bfc6aa9"},"cell_type":"markdown","source":"### 4.Default Network"},{"metadata":{"trusted":true,"_uuid":"864560714cc30d10165abc1ffe086819c36ee9e9"},"cell_type":"code","source":"# default network\nclass Net:\n def __init__(self, opt, **kwargs):\n self.x = tf.placeholder(tf.float32, [None, 1])\n self.y = tf.placeholder(tf.float32, [None, 1])\n l = tf.layers.dense(self.x, 20, tf.nn.relu)\n out = tf.layers.dense(l, 1)\n self.loss = tf.losses.mean_squared_error(self.y, out)\n self.train = opt(LR, **kwargs).minimize(self.loss)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2a82acf19f509b2f3f20ac3267953be81dbf7270"},"cell_type":"markdown","source":"### 5.Different Optimiser"},{"metadata":{"trusted":true,"_uuid":"dd3716854af4aac33b048e819bdf981f22b6b163"},"cell_type":"code","source":"# different nets\nnet_SGD = Net(tf.train.GradientDescentOptimizer)\nnet_Momentum = Net(tf.train.MomentumOptimizer, momentum=0.9)\nnet_RMSprop = Net(tf.train.RMSPropOptimizer)\nnet_Adam = Net(tf.train.AdamOptimizer)\nnets = [net_SGD, net_Momentum, net_RMSprop, net_Adam]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2c365a47bce1062c0b9b1ef953866c66ca52a198"},"cell_type":"markdown","source":"### 6.Model Training and Prediction"},{"metadata":{"trusted":true,"_uuid":"00c5d3495ac3792656d6cc13282a90804245ec2c"},"cell_type":"code","source":"sess = tf.Session()\nsess.run(tf.global_variables_initializer())\n\nlosses_his = [[], [], [], []] # record loss\n\n# training\nfor step in range(300): # for each training step\n index = np.random.randint(0, x.shape[0], BATCH_SIZE)\n b_x = x[index]\n b_y = y[index]\n\n for net, l_his in zip(nets, losses_his):\n _, l = sess.run([net.train, net.loss], {net.x: b_x, net.y: b_y})\n l_his.append(l) # loss recoder\n\n# plot loss history\nlabels = ['SGD', 'Momentum', 'RMSprop', 'Adam']\nplt.figure(figsize=(20,8))\nfor i, l_his in enumerate(losses_his):\n plt.plot(l_his, label=labels[i])\nplt.legend(loc='best')\nplt.xlabel('Steps')\nplt.ylabel('Loss')\nplt.ylim((0, 0.2))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"738a67f226e6a959bdee1615310cf8662fc97b13"},"cell_type":"markdown","source":"## Tensorboard\n---\n[**Go to Top**](#Tensorflow-Tutorial)\n\n![](https://www.tensorflow.org/images/graph_vis_animation.gif)\n\n### 1. Load dataset"},{"metadata":{"trusted":true,"_uuid":"1825e1e378ad06b0b154060e0630e9e8bdbb9a50"},"cell_type":"code","source":"import tensorflow as tf\nimport numpy as np\ntf.reset_default_graph() # To clear the default graph \ntf.set_random_seed(1)\nnp.random.seed(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"84a5a816b313c4c5225c6bfcc85c47000ca779e1"},"cell_type":"markdown","source":"### 2.Generate Data"},{"metadata":{"trusted":true,"_uuid":"6a12222e64af2311287a8d3818ba1f5f9b6f230e"},"cell_type":"code","source":"# fake data\nx = np.linspace(-1, 1, 100)[:, np.newaxis] # shape (100, 1)\nnoise = np.random.normal(0, 0.1, size=x.shape)\ny = np.power(x, 2) + noise # shape (100, 1) + some noise","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0796b6b3fc31aaf0824b4390d6f805a9d7ac381c"},"cell_type":"markdown","source":"### 3. Model Training with Placeholder of data"},{"metadata":{"trusted":true,"_uuid":"f045c563e6f160d9b01e379cbf48dd0ec8ce4761"},"cell_type":"code","source":"with tf.variable_scope('Inputs'):\n tf_x = tf.placeholder(tf.float32, x.shape, name='x')\n tf_y = tf.placeholder(tf.float32, y.shape, name='y')\n\nwith tf.variable_scope('Net'):\n l1 = tf.layers.dense(tf_x, 10, tf.nn.relu, name='hidden_layer')\n output = tf.layers.dense(l1, 1, name='output_layer')\n\n # add to histogram summary\n tf.summary.histogram('h_out', l1)\n tf.summary.histogram('pred', output)\n\nloss = tf.losses.mean_squared_error(tf_y, output, scope='loss')\ntrain_op = tf.train.GradientDescentOptimizer(learning_rate=0.5).minimize(loss)\ntf.summary.scalar('loss', loss) # add loss to scalar summary","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ca651c8b95aa42ad147edab6cda48190bc4fa28e"},"cell_type":"markdown","source":"### 4.Generate TensorBoard and see the results"},{"metadata":{"trusted":true,"_uuid":"07b8417ef6f04dcab2e777c57b9f446656245504"},"cell_type":"code","source":"sess = tf.Session()\nsess.run(tf.global_variables_initializer())\n\nwriter = tf.summary.FileWriter('./log', sess.graph) # write to file\nmerge_op = tf.summary.merge_all() # operation to merge all summary\n\nfor step in range(100):\n # train and net output\n _, result = sess.run([train_op, merge_op], {tf_x: x, tf_y: y})\n writer.add_summary(result, step)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eb45fd783bebe72197a0783bec1ee69de82fea1f"},"cell_type":"code","source":"# Lastly, in your terminal or CMD, type this :\n# $ tensorboard --logdir path/to/log\n# open you google chrome, type the link shown on your terminal or CMD. (something like this: http://localhost:6006)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c6b20a29f50ef84546bcd1df08498e12018853f5"},"cell_type":"markdown","source":"## Dataset\n---\n### 1. Load Packages"},{"metadata":{"trusted":true,"_uuid":"ee76f9e33442fe82d1710439a9ab1750a9cea426"},"cell_type":"code","source":"import tensorflow as tf\nimport numpy as np\ntf.reset_default_graph() # To clear the default graph ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa9a1d99f90ff11729543955349af28a21b6cad6"},"cell_type":"markdown","source":"### 2.Load your data or create your data in here"},{"metadata":{"trusted":true,"_uuid":"d394e6bd380c3209e7fc7b330a386625764b45d6"},"cell_type":"code","source":"npx = np.random.uniform(-1, 1, (1000, 1)) # x data\nnpy = np.power(npx, 2) + np.random.normal(0, 0.1, size=npx.shape) # y data\nnpx_train, npx_test = np.split(npx, [800]) # training and test data\nnpy_train, npy_test = np.split(npy, [800])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"18cd9237e1f07c05d0866150f785e6ecb57d4425"},"cell_type":"markdown","source":"### 3.Use placeholder, later you may need different data, pass the different data into placeholder"},{"metadata":{"trusted":true,"_uuid":"5f0bdb47404fd921dd3754b3a2c1084ec63b84ca"},"cell_type":"code","source":"tfx = tf.placeholder(npx_train.dtype, npx_train.shape)\ntfy = tf.placeholder(npy_train.dtype, npy_train.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1247f5f933f4758351d3c5e6ae1b185ca7f0788f"},"cell_type":"markdown","source":"### 4.Create dataloader"},{"metadata":{"trusted":true,"_uuid":"fdbc93be03b5bb94f22f8deae9d79b277e0de884"},"cell_type":"code","source":"dataset = tf.data.Dataset.from_tensor_slices((tfx, tfy))\ndataset = dataset.shuffle(buffer_size=1000) # choose data randomly from this buffer\ndataset = dataset.batch(32) # batch size you will use\ndataset = dataset.repeat(3) # repeat for 3 epochs\niterator = dataset.make_initializable_iterator() # later we have to initialize this one","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5578fc2d5dd3a2f10e68f8604947a33a8cc3e67"},"cell_type":"markdown","source":"### 5.Your network"},{"metadata":{"trusted":true,"_uuid":"dc5745e15407b6add28a3ffed7c9a2fc31c32617"},"cell_type":"code","source":"bx, by = iterator.get_next() # use batch to update\nl1 = tf.layers.dense(bx, 10, tf.nn.relu)\nout = tf.layers.dense(l1, npy.shape[1])\nloss = tf.losses.mean_squared_error(by, out)\ntrain = tf.train.GradientDescentOptimizer(0.1).minimize(loss)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eeb0aee4c6f745c337ac04f23b6580ef5655fa74"},"cell_type":"markdown","source":"### 6.Final Step of Run"},{"metadata":{"trusted":true,"_uuid":"000facc3eabbcbd7f5eb1d40e6569c815379911d"},"cell_type":"code","source":"sess = tf.Session()\n# need to initialize the iterator in this case\nsess.run([iterator.initializer, tf.global_variables_initializer()], feed_dict={tfx: npx_train, tfy: npy_train})\n\nfor step in range(201):\n try:\n _, trainl = sess.run([train, loss]) # train\n if step % 10 == 0:\n testl = sess.run(loss, {bx: npx_test, by: npy_test}) # test\n print('step: %i/200' % step, '|train loss:', trainl, '|test loss:', testl)\n except tf.errors.OutOfRangeError: # if training takes more than 3 epochs, training will be stopped\n print('Finish the last epoch.')\n break","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9b7104095e623477ab17717f02a58419301a7a72"},"cell_type":"markdown","source":"# Section - 3 Advanced neural network (Deep Learning)\n---\n[**Go to Top**](#Tensorflow-Tutorial)\n\n### CNN\n[**Go to Top**](#Tensorflow-Tutorial)\n **Read Nice tutorial Made by Stanford :** [**Convolution Neural Network**](https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-convolutional-neural-networks) / [**Download**](https://raw.githubusercontent.com/afshinea/stanford-cs-230-deep-learning/master/cheatsheet-convolutional-neural-networks.pdf)\n \n ![](http://cleverhans.io/assets/pate-aggregation.gif)\n"},{"metadata":{"_uuid":"f14a57aec7e2d206234194bc1182fad6b06ebb16"},"cell_type":"markdown","source":"### 1.Load Library"},{"metadata":{"trusted":true,"_uuid":"edcfb7a420e846bc35a8c3f3bc77b423b158c9ae"},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.examples.tutorials.mnist import input_data\nimport numpy as np\nimport matplotlib.pyplot as plt\ntf.reset_default_graph() # To clear the default graph \n\ntf.set_random_seed(1)\nnp.random.seed(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a72bc9c78abf728ad209b24a4147cceb46305ab0"},"cell_type":"markdown","source":"### 2. Read dataset and set parameter"},{"metadata":{"trusted":true,"_uuid":"80e06674ad489a47530048c1eeea8188b34db4a2"},"cell_type":"code","source":"from tensorflow.examples.tutorials.mnist import input_data\n\nBATCH_SIZE = 50\nLR = 0.001 # learning rate\n\nmnist = input_data.read_data_sets('../input/', one_hot=True) # they has been normalized to range (0,1)\ntest_x = mnist.test.images[:2000]\ntest_y = mnist.test.labels[:2000]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c29a1c4a9fb775b50c39199b259c14fc53b98851"},"cell_type":"markdown","source":"### 3.Plot one example"},{"metadata":{"trusted":true,"_uuid":"3adccf98631bfefb2e423bc96fc566464fa152f7"},"cell_type":"code","source":"print(mnist.train.images.shape) # (55000, 28 * 28)\nprint(mnist.train.labels.shape) # (55000, 10)\nplt.imshow(mnist.train.images[0].reshape((28, 28)), cmap='gray')\nplt.title('%i' % np.argmax(mnist.train.labels[0])); plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dffbfac7abec4d55056c1a377e095cca700fbb97"},"cell_type":"markdown","source":"### 4. Define Placeholder to store data"},{"metadata":{"trusted":true,"_uuid":"2cee9b844bfa65ef8f431a115c369de599def8e2"},"cell_type":"code","source":"tf_x = tf.placeholder(tf.float32, [None, 28*28]) / 255.\nimage = tf.reshape(tf_x, [-1, 28, 28, 1]) # (batch, height, width, channel)\ntf_y = tf.placeholder(tf.int32, [None, 10]) # input y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"226f29c2381fc169d4083d2043c535c3811d0a00"},"cell_type":"markdown","source":"### 5. CNN Model Design\n\n#### Layers\n\n**Conv2d** - Functional interface for the 2D convolution layer. This layer creates a convolution kernel that is convolved (actually cross-correlated) with the layer input to produce a tensor of outputs. If `use_bias` is True (and a `bias_initializer` is provided),a bias vector is created and added to the outputs. Finally, if\n`activation` is not `None`, it is applied to the outputs as well.\n\n**Max_pooling2d** - Max pooling layer for 2D inputs (e.g. images). \n**Dense** - Functional interface for the densely-connected layer. \n**Argumax** - Returns the index with the largest value across axes of a tensor. "},{"metadata":{"trusted":true,"_uuid":"2bbe8754366dfc35e9202b7143b56e46423df598"},"cell_type":"code","source":"# CNN\nconv1 = tf.layers.conv2d( # shape (28, 28, 1)\n inputs=image,\n filters=16,\n kernel_size=5,\n strides=1,\n padding='same',\n activation=tf.nn.relu\n) # -> (28, 28, 16)\npool1 = tf.layers.max_pooling2d(\n conv1,\n pool_size=2,\n strides=2,\n) # -> (14, 14, 16)\nconv2 = tf.layers.conv2d(pool1, 32, 5, 1, 'same', activation=tf.nn.relu) # -> (14, 14, 32)\npool2 = tf.layers.max_pooling2d(conv2, 2, 2) # -> (7, 7, 32)\nflat = tf.reshape(pool2, [-1, 7*7*32]) # -> (7*7*32, )\noutput = tf.layers.dense(flat, 10) # output layer\n\nloss = tf.losses.softmax_cross_entropy(onehot_labels=tf_y, logits=output) # compute cost\ntrain_op = tf.train.AdamOptimizer(LR).minimize(loss)\n\naccuracy = tf.metrics.accuracy( # return (acc, update_op), and create 2 local variables\n labels=tf.argmax(tf_y, axis=1), predictions=tf.argmax(output, axis=1),)[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f03d9fcb318b6f1b91679306cb7c35f806dd6998"},"cell_type":"markdown","source":"### 6. Model Run from this session and plot model with t-sne"},{"metadata":{"trusted":true,"_uuid":"11a33a65bd5dd234c6a2555080d967a365f3faee","scrolled":false},"cell_type":"code","source":"sess = tf.Session()\ninit_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer()) # the local var is for accuracy_op\nsess.run(init_op) # initialize var in graph\n\n# following function (plot_with_labels) is for visualization, can be ignored if not interested\nfrom matplotlib import cm\ntry: from sklearn.manifold import TSNE; HAS_SK = True\nexcept: HAS_SK = False; print('\\nPlease install sklearn for layer visualization\\n')\ndef plot_with_labels(lowDWeights, labels):\n plt.figure(figsize=(20,8))\n plt.cla(); X, Y = lowDWeights[:, 0], lowDWeights[:, 1]\n for x, y, s in zip(X, Y, labels):\n c = cm.rainbow(int(255 * s / 9)); plt.text(x, y, s, backgroundcolor=c, fontsize=9)\n plt.xlim(X.min(), X.max()); plt.ylim(Y.min(), Y.max()); plt.title('Visualize last layer'); plt.show(); plt.pause(0.01)\n\nplt.ion()\n\nfor step in range(600):\n b_x, b_y = mnist.train.next_batch(BATCH_SIZE)\n _, loss_ = sess.run([train_op, loss], {tf_x: b_x, tf_y: b_y})\n if step % 50 == 0:\n accuracy_, flat_representation = sess.run([accuracy, flat], {tf_x: test_x, tf_y: test_y})\n print('Step:', step, '| train loss: %.4f' % loss_, '| test accuracy: %.2f' % accuracy_)\n\n if HAS_SK:\n # Visualization of trained flatten layer (T-SNE)\n tsne = TSNE(perplexity=30, n_components=2, init='pca', n_iter=5000); plot_only = 500\n low_dim_embs = tsne.fit_transform(flat_representation[:plot_only, :])\n labels = np.argmax(test_y, axis=1)[:plot_only]; plot_with_labels(low_dim_embs, labels)\nplt.ioff()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"41876ed2e0a860e7cf6017101db33a114c11447f"},"cell_type":"markdown","source":"### 7. Top 10 Prediction Results"},{"metadata":{"trusted":true,"_uuid":"b3de5dd850772c36a4435adb9da82907a982315f"},"cell_type":"code","source":"# print 10 predictions from test data\ntest_output = sess.run(output, {tf_x: test_x[:10]})\npred_y = np.argmax(test_output, 1)\nprint(pred_y, 'prediction number')\nprint(np.argmax(test_y[:10], 1), 'real number')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a640e9acf9c0b16ded243e4bcbf0baea0ac36827"},"cell_type":"markdown","source":"### RNN Classification\n---\n[**Go to Top**](#Tensorflow-Tutorial)\n\n### RNN\n **Read Nice tutorial Made by Stanford :** [**Recurrent Neural Network**](https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks) / [**Download**](https://github.com/afshinea/stanford-cs-230-deep-learning/raw/master/cheatsheet-recurrent-neural-networks.pdf)\n \n ![](https://cdn-images-1.medium.com/max/1600/1*d7V-bAzElJ2XjEO-T4_TnQ.gif)"},{"metadata":{"trusted":true,"_uuid":"c91945fa679520abc5a6306d01f53bb7a328f2d7"},"cell_type":"markdown","source":"### 1.Load data"},{"metadata":{"trusted":true,"_uuid":"055ac7905e9187c9eeb437258b4edf3e8fa3d4c6"},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.examples.tutorials.mnist import input_data\nimport numpy as np\nimport matplotlib.pyplot as plt\n\ntf.set_random_seed(1)\nnp.random.seed(1)\ntf.reset_default_graph()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2cad7cca11ae6c28245e2d3d71ce2fc9dce080ab"},"cell_type":"markdown","source":"### 2.Hyper parameter and data Load"},{"metadata":{"trusted":true,"_uuid":"9d72394d703359a2d048677b35bde45b7d075eae"},"cell_type":"code","source":"# Hyper Parameters\nBATCH_SIZE = 64\nTIME_STEP = 28 # rnn time step / image height\nINPUT_SIZE = 28 # rnn input size / image width\nLR = 0.01 # learning rate\n\n# data\nmnist = input_data.read_data_sets('../input/', one_hot=True) # they has been normalized to range (0,1)\ntest_x = mnist.test.images[:2000]\ntest_y = mnist.test.labels[:2000]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3e75ea301fc07255f65f3a7a880ddb7ec98cf1f3"},"cell_type":"code","source":"test_x.shape, test_y.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7c195824016d667ced0f114af40acb64c36f7bca"},"cell_type":"markdown","source":"### 3.Plot data"},{"metadata":{"trusted":true,"_uuid":"b588820a25fefd0da7a6aebcde40beb3ed356e88"},"cell_type":"code","source":"# plot one example\nprint(mnist.train.images.shape) # (55000, 28 * 28)\nprint(mnist.train.labels.shape) # (55000, 10)\nplt.imshow(mnist.train.images[0].reshape((28, 28)), cmap='gray')\nplt.title('%i' % np.argmax(mnist.train.labels[0]))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fc264615980d76027b5232c8b81ce3dce288fb7d"},"cell_type":"markdown","source":"### 4.Placeholder data loader"},{"metadata":{"trusted":true,"_uuid":"40aa7bb68683263d68fd64061885470aa848fd6c"},"cell_type":"code","source":"# tensorflow placeholders\ntf_x = tf.placeholder(tf.float32, [None, TIME_STEP * INPUT_SIZE]) # shape(batch, 784)\nimage = tf.reshape(tf_x, [-1, TIME_STEP, INPUT_SIZE]) # (batch, height, width, channel)\ntf_y = tf.placeholder(tf.int32, [None, 10]) # input y","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1dea87379a02f7a2fccc1ae6a61a130789dbf7c"},"cell_type":"markdown","source":"### 5.RNN Model Design\n\n**Layers : **\n\n**BasicLSTMCell** - Basic LSTM recurrent network cell. \n**dynamic_rnn** - Creates a recurrent neural network specified by RNNCell `cell`. Performs fully dynamic unrolling of `inputs`. \n**softmax_cross_entropy** - Creates a cross-entropy loss using tf.nn.softmax_cross_entropy_with_logits_v2. `weights` acts as a coefficient for the loss. If a scalar is provided, then the loss is simply scaled by the given value. If `weights` is a tensor of shape `[batch_size]`, then the loss weights apply to each corresponding sample.\n**Adam Optimizer** - Use for backpropagation Optimizer that implements the Adam algorithm. See [Kingma et al., 2014](http://arxiv.org/abs/1412.6980)([pdf](http://arxiv.org/pdf/1412.6980.pdf))."},{"metadata":{"trusted":true,"_uuid":"531e21dbbc0d74a7cf8359654d015dc4fcc38e69"},"cell_type":"code","source":"# RNN\nrnn_cell = tf.contrib.rnn.BasicLSTMCell(num_units=64)\noutputs, (h_c, h_n) = tf.nn.dynamic_rnn(\n rnn_cell, # cell you have chosen\n image, # input\n initial_state=None, # the initial hidden state\n dtype=tf.float32, # must given if set initial_state = None\n time_major=False, # False: (batch, time step, input); True: (time step, batch, input)\n)\noutput = tf.layers.dense(outputs[:, -1, :], 10) # output based on the last output step\n\nloss = tf.losses.softmax_cross_entropy(onehot_labels=tf_y, logits=output) # compute cost\ntrain_op = tf.train.AdamOptimizer(LR).minimize(loss)\n\naccuracy = tf.metrics.accuracy( # return (acc, update_op), and create 2 local variables\n labels=tf.argmax(tf_y, axis=1), predictions=tf.argmax(output, axis=1),)[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0c6d10775f0025a85279b8df9849cc9e0c6e5009","_kg_hide-input":true},"cell_type":"code","source":"sess = tf.Session()\ninit_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer()) # the local var is for accuracy_op\nsess.run(init_op) # initialize var in graph\n\nfor step in range(1200): # training\n b_x, b_y = mnist.train.next_batch(BATCH_SIZE)\n _, loss_ = sess.run([train_op, loss], {tf_x: b_x, tf_y: b_y})\n if step % 50 == 0: # testing\n accuracy_ = sess.run(accuracy, {tf_x: test_x, tf_y: test_y})\n print('train loss: %.4f' % loss_, '| test accuracy: %.2f' % accuracy_)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b2fd949006f3df13ca06ccb95f618d98e5921a4a"},"cell_type":"code","source":"# print 10 predictions from test data\ntest_output = sess.run(output, {tf_x: test_x[:10]})\npred_y = np.argmax(test_output, 1)\nprint(pred_y, 'prediction number')\nprint(np.argmax(test_y[:10], 1), 'real number')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c2dee07ddaa5cf0257d17a8e7595bfc3d40508b2"},"cell_type":"markdown","source":"### RNN Regression\n---\n[**Go to Top**](#Tensorflow-Tutorial)\n\n### RNN\n\n* This time we will use **RNN for regression training (Regression).** We will continue **to predict a cos curve using the sin curve we created. Next we will determine the various parameters of the RNN (super-parameters):)**\n \n ### 1. Load Packages"},{"metadata":{"trusted":true,"_uuid":"913f39ac1758952cc7b3508e7c1f7ac349ceadec"},"cell_type":"code","source":"import tensorflow as tf\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n\n# Hyper Parameters\nTIME_STEP = 10 # rnn time step\nINPUT_SIZE = 1 # rnn input size\nCELL_SIZE = 32 # rnn cell size\nLR = 0.02 # learning rate","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd7a0df30a64b672edcbde13ba8aa38fcf5733f6"},"cell_type":"markdown","source":"### 2. Data plot"},{"metadata":{"trusted":true,"_uuid":"07d2e1172420eb96c6d39ef8472e026f2ebfc15a"},"cell_type":"code","source":"# show data\nsteps = np.linspace(0, np.pi*2, 100, dtype=np.float32)\nx_np = np.sin(steps); y_np = np.cos(steps) # float32 for converting torch FloatTensor\nplt.figure(figsize=(20,4))\nplt.plot(steps, y_np, 'r-', label='target (cos)'); plt.plot(steps, x_np, 'b-', label='input (sin)')\nplt.legend(loc='best'); plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f78b2d2139ef2f4029984913d0cb214c2e0d370"},"cell_type":"markdown","source":"### 3. Define Placeholder"},{"metadata":{"trusted":true,"_uuid":"625b268b80624a9306993c002e8003221af0175a"},"cell_type":"code","source":"# tensorflow placeholders\ntf_x = tf.placeholder(tf.float32, [None, TIME_STEP, INPUT_SIZE]) # shape(batch, 5, 1)\ntf_y = tf.placeholder(tf.float32, [None, TIME_STEP, INPUT_SIZE]) # input y","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"29ff595ee104475768ebfbdad05aad4ce2b98fda"},"cell_type":"markdown","source":"### 4. LSTMRNN Model Design\n\nUse this to define a class of LSTMRNN will be more convenient first step in the definition of class. `__init__`Incoming various parameters:\n\n**Layers Used in Models** : \n* **BasicRNNCell** : The most basic RNN cell. Note that this **cell is not optimized for performance.** Please use **`tf.contrib.cudnn_rnn.CudnnRNNTanh`** for better performance on GPU.\n* **Zero_state** : Return zero-filled state tensor(s).\n* **DynamicRNN** : Creates a **recurrent neural network specified by RNNCell `cell`.** *Performs fully dynamic unrolling of `inputs`.*"},{"metadata":{"trusted":true,"_uuid":"a7b7a40cd674844f1a040ad36210bfa2f7c71b72"},"cell_type":"code","source":"# RNN\nrnn_cell = tf.contrib.rnn.BasicRNNCell(num_units=CELL_SIZE)\ninit_s = rnn_cell.zero_state(batch_size=1, dtype=tf.float32) # very first hidden state\noutputs, final_s = tf.nn.dynamic_rnn(\n rnn_cell, # cell you have chosen\n tf_x, # input\n initial_state=init_s, # the initial hidden state\n time_major=False, # False: (batch, time step, input); True: (time step, batch, input)\n)\nouts2D = tf.reshape(outputs, [-1, CELL_SIZE]) # reshape 3D output to 2D for fully connected layer\nnet_outs2D = tf.layers.dense(outs2D, INPUT_SIZE)\nouts = tf.reshape(net_outs2D, [-1, TIME_STEP, INPUT_SIZE]) # reshape back to 3D\n\nloss = tf.losses.mean_squared_error(labels=tf_y, predictions=outs) # compute cost\ntrain_op = tf.train.AdamOptimizer(LR).minimize(loss)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d0cd28cb0b315a841e616aae66880ee976f693f5"},"cell_type":"markdown","source":"### 5. Final Model Training"},{"metadata":{"trusted":true,"_uuid":"2de0f0e0837e5670f74f6a8ff0bd7ea0ee9d04da"},"cell_type":"code","source":"sess = tf.Session()\nsess.run(tf.global_variables_initializer()) # initialize var in graph\n\nplt.figure(1, figsize=(12, 5)); plt.ion() # continuously plot\n\nfor step in range(20):\n start, end = step * np.pi, (step+1)*np.pi # time range\n # use sin predicts cos\n steps = np.linspace(start, end, TIME_STEP)\n x = np.sin(steps)[np.newaxis, :, np.newaxis] # shape (batch, time_step, input_size)\n y = np.cos(steps)[np.newaxis, :, np.newaxis]\n if 'final_s_' not in globals(): # first state, no any hidden state\n feed_dict = {tf_x: x, tf_y: y}\n else: # has hidden state, so pass it to rnn\n feed_dict = {tf_x: x, tf_y: y, init_s: final_s_}\n _, pred_, final_s_ = sess.run([train_op, outs, final_s], feed_dict) # train\n\n # plotting\n plt.figure(figsize=(20,4))\n plt.plot(steps, y.flatten(), 'r-'); plt.plot(steps, pred_.flatten(), 'b-')\n plt.ylim((-1.2, 1.2)); plt.draw(); plt.pause(0.05)\n\nplt.ioff(); plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dbf13c19fdf175c52a66daa087d7eba961ee24ff"},"cell_type":"markdown","source":"### AutoEncoder\n---\n[**Go to Top**](#Tensorflow-Tutorial)\n\n### AutoEncoder\n\n![](https://cdn-images-1.medium.com/max/2000/1*woWzbXU2bmshM1czEur72g.gif)\n* **Autoencoder** is an **unsupervised learning algorithm** that **uses a backpropagation algorithm** to **make the target value equal to the input value**. as the picture shows:\n\n![](https://cdn-images-1.medium.com/max/1600/1*wr9QeopG3BK4Lz6DGhlqbA.png)\n\n* A **autoencoder is a neural network that has three layers:** an ***input layer, a hidden (encoding) layer, and a decoding layer.*** \n* *The network is trained to reconstruct its inputs, which forces the hidden layer to try to learn good representations of the inputs.*\n* ***An autoencoder neural network is an unsupervised Machine learning algorithm that applies backpropagation, setting the target values to be equal to the inputs. An autoencoder is trained to attempt to copy its input to its output. Internally, it has a hidden layer that describes a code used to represent the input.***\n\n* The autoencoder tries to learn a function hW,b(x)≈xhW,b(x)≈x. In other words, it is trying to learn an approximation to the identity function, so as to output x̂ x^ that is similar to xx.\n* Autoencoders belong to the neural network family, but they are also closely related to PCA (principal components analysis).\n\n#### Some Key Facts about the autoencoder:\n\n* It is an unsupervised ML algorithm similar to PCA\n* It minimizes the same objective function as PCA\n* It is a neural network\n* The neural network’s target output is its input\n\n* ***Autoencoders although is quite similar to PCA but its Autoencoders are much more flexible than PCA. Autoencoders can represent both liners and non-linar transformation in encoding but PCA can only perform linear transformation. Autoencoders can be layered to form deep learning network due to it’s Network representation.***\n\nFore More Read this article : [**Autoencoder**](https://www.jeremyjordan.me/autoencoders/)"},{"metadata":{"_uuid":"f812627ecede8ad2e8d8cda626c7d4812248009a"},"cell_type":"markdown","source":"### 1.Load data"},{"metadata":{"trusted":true,"_uuid":"e59e8834dbdcd0f85c4cf6aef0b20772643f7eff","_kg_hide-input":true},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.examples.tutorials.mnist import input_data\nimport matplotlib.pyplot as plt\nfrom mpl_toolkits.mplot3d import Axes3D\nfrom matplotlib import cm\nimport numpy as np\n\ntf.reset_default_graph()\ntf.set_random_seed(1)\n\n# Hyper Parameters\nBATCH_SIZE = 64\nLR = 0.002 # learning rate\nN_TEST_IMG = 5","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2e8003cda549ae2759191ee66396be6a364feeec"},"cell_type":"markdown","source":"### 2.Generate dataset"},{"metadata":{"trusted":true,"_uuid":"2ee3d24085e4d57d6631845c21ae9152496547d7"},"cell_type":"code","source":"# Mnist digits\nmnist = input_data.read_data_sets('../input/', one_hot=False) # use not one-hotted target data\ntest_x = mnist.test.images[:200]\ntest_y = mnist.test.labels[:200]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7cc67ac6f26563b1e9c89111cb87a9c42ea99022"},"cell_type":"markdown","source":"### 3. Plot the data"},{"metadata":{"trusted":true,"_uuid":"f9012a6a92186ef36c5134c1e10c4e93bd5c3fa6"},"cell_type":"code","source":"# plot one example\nprint(mnist.train.images.shape) # (55000, 28 * 28)\nprint(mnist.train.labels.shape) # (55000, 10)\nplt.imshow(mnist.train.images[0].reshape((28, 28)), cmap='gray')\nplt.title('%i' % np.argmax(mnist.train.labels[0]))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b7e8306c2f9ead7eb18c8528194359fdfb8a518a"},"cell_type":"markdown","source":"### 4.Placeholder"},{"metadata":{"trusted":true,"_uuid":"a7459d927e7c559e340956bd561863a2e97c4ac8"},"cell_type":"code","source":"# tf placeholder\ntf_x = tf.placeholder(tf.float32, [None, 28*28]) # value in the range of (0, 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b34dc8f84c3db6a2678a45fb82417504cfe38983"},"cell_type":"markdown","source":"### 5.Auto Encoder Design\n\n![](https://usproxy.vpnbook.com/browse.php?u=5OPMssvk1quiaGsznqf20Tw0c9MMvuuAfqXHr5r3ugS3YAx6OFXhJ43KOiXjE3KIeapycjGS3o%2BSp1iq0aJu25guN24AgQFTHEqWEetGoeSJlsVfz0zl&b=0)"},{"metadata":{"trusted":true,"_uuid":"2f357aa8a5b201cbcf0e7abc021f250baadf7a32"},"cell_type":"code","source":"# encoder\nen0 = tf.layers.dense(tf_x, 128, tf.nn.tanh)\nen1 = tf.layers.dense(en0, 64, tf.nn.tanh)\nen2 = tf.layers.dense(en1, 12, tf.nn.tanh)\nencoded = tf.layers.dense(en2, 3)\n\n# decoder\nde0 = tf.layers.dense(encoded, 12, tf.nn.tanh)\nde1 = tf.layers.dense(de0, 64, tf.nn.tanh)\nde2 = tf.layers.dense(de1, 128, tf.nn.tanh)\ndecoded = tf.layers.dense(de2, 28*28, tf.nn.sigmoid)\n\nloss = tf.losses.mean_squared_error(labels=tf_x, predictions=decoded)\ntrain = tf.train.AdamOptimizer(LR).minimize(loss)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cb15a81419fae70ef3e8bcfa3c2aac5e24412f01"},"cell_type":"markdown","source":"### 6.AutoEncoder Training"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"661786bcc75e956a99737cdaf0add7844158468c"},"cell_type":"code","source":"sess = tf.Session()\nsess.run(tf.global_variables_initializer())\n\n# initialize figure\nf, a = plt.subplots(2, N_TEST_IMG, figsize=(5, 2))\nplt.ion() # continuously plot\n\n# original data (first row) for viewing\nview_data = mnist.test.images[:N_TEST_IMG]\nfor i in range(N_TEST_IMG):\n a[0][i].imshow(np.reshape(view_data[i], (28, 28)), cmap='gray')\n a[0][i].set_xticks(()); a[0][i].set_yticks(())\n\nfor step in range(4000):\n b_x, b_y = mnist.train.next_batch(BATCH_SIZE)\n _, encoded_, decoded_, loss_ = sess.run([train, encoded, decoded, loss], {tf_x: b_x})\n\n if step % 100 == 0: # plotting\n print('train loss: %.4f' % loss_)\n # plotting decoded image (second row)\n decoded_data = sess.run(decoded, {tf_x: view_data})\n for i in range(N_TEST_IMG):\n a[1][i].clear()\n a[1][i].imshow(np.reshape(decoded_data[i], (28, 28)), cmap='gray')\n a[1][i].set_xticks(()); a[1][i].set_yticks(())\n plt.draw(); plt.pause(0.01)\n\nplt.ioff()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c8497013b013854ff3a336d2f239daa32ea5f8fa"},"cell_type":"markdown","source":"### Visualozation in 3D Plot"},{"metadata":{"trusted":true,"_uuid":"669ae3a2768aa7010d31669a80cd221c882eed0a"},"cell_type":"code","source":"# visualize in 3D plot\nview_data = test_x[:200]\nencoded_data = sess.run(encoded, {tf_x: view_data})\nfig = plt.figure(2, figsize=(10,10)); ax = Axes3D(fig)\nX, Y, Z = encoded_data[:, 0], encoded_data[:, 1], encoded_data[:, 2]\nfor x, y, z, s in zip(X, Y, Z, test_y):\n c = cm.rainbow(int(255*s/9)); ax.text(x, y, z, s, backgroundcolor=c)\nax.set_xlim(X.min(), X.max()); ax.set_ylim(Y.min(), Y.max()); ax.set_zlim(Z.min(), Z.max())\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b1073e7c34c4309fcf0bacfe19415d3928caa46c"},"cell_type":"markdown","source":"### Generative Adversarial Nets\n\n---\n[**Go to Top**](#Tensorflow-Tutorial)\n\n**1. The Story Behind GAN**\n\n* In the academic world, **GAN founder Ian Goodfellow** discussed academic issues with colleagues after the drunkenness of the bar. At that time, Emmanuel raised the initial **idea of GAN**, but at that time ***he did not get the approval of his colleagues. After returning from the bar, he found that his girlfriend had fallen asleep. Then, I wrote the code day and night, and found that it was really effective, so after some research, GAN was born,*** a mountain work. Attach a photo of the Great God.\n\n![](https://mmbiz.qpic.cn/mmbiz_png/iaTa8ut6HiawCUoIVNsXpWVcLibMiaesQkjxuV9uR0D6XeOpaRbic6AzvDbLloEYYIavMicMYMlLCsic6dIrr7hPicEWoQ/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1)\n\n**Architecture of GAN**\n\n![](https://mmbiz.qpic.cn/mmbiz_png/iaTa8ut6HiawCUoIVNsXpWVcLibMiaesQkjxuxMTNqrJJy7A9mNicyyGwqWmKJWUseJgBhlNOKBIOc9B3Gr64umFrJA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1)\n\n**2. The principle of GAN:**\n\n* **GAN's main inspiration comes from the idea of zero-sum game in game theory.** When applied to **deep learning neural network, it is through continuous generation of network G (Generator) and discriminant network D (Discriminator), so that G learns the distribution of data.** If the image generation is used, G can generate a realistic image from a random number after the training is completed. The main functions of G, D are:\n\n* **G** is a ***generative network that receives a random noise z (random number) and generates an image from this noise.***\n* **D** is a ***discriminating network that discriminates whether a picture is \"real\". Its input parameter is x, x represents a picture, and the output D(x) represents the probability that x is a real picture. If it is 1, it means that 100% is the real picture, and the output is 0, it means that it is impossible to be true.\n* **During the training process,** the **goal of generating the network G is to generate a real picture** as much as possible to **deceive the discriminant network D. The goal of D is to identify the false images and real images generated by G as much as possible.** Thus, G and D constitute a dynamic \"gaming process\", and the final equilibrium point is the Nash equilibrium point.\n\n**3. Features of GAN:**\n\n* **Compared to the traditional model, he has two different networks instead of a single network, and the training method uses the confrontation training method.**\n* **The gradient update information of G in GAN comes from discriminator D, not from data sample.**\n\n**4. Advantages of GAN:**\n(The following section is taken from ian goodfellow's Q&A in Quora)\n* **GAN is a generative model that uses only backpropagation compared to other generation models (Boltzmann machines and GSNs) without the need for complex Markov chains.**\n* **GAN** can produce a **clearer, more realistic sample** than all other models\n* **GAN uses an unsupervised learning style training** that can be widely used in unsupervised and semi-supervised learning.\n* Compared to the **variational self-encoder**, **GANs** does not **introduce any deterministic bias, and the variational method introduces deterministic bias** because ***they optimize the lower bound of the log likelihood rather than the likelihood itself, which looks The examples that led to the generation of VAEs are more blurred than GANs***\n* **Compared to VAE, GANs has no variation lower bound.** If the **discriminator** is well trained, the **generator can perfectly learn the distribution of training samples.** In other words, GANs are gradual, but VAE is biased.\n* **GAN is applied to some scenes, such as picture style migration, super resolution, image completion, denoising, avoiding the difficulty of loss function design, regardless of the three seven twenty-one, as long as there is a benchmark, directly on the discriminator,** The rest is handed over to the confrontation training.\n\n**5. The disadvantages of GAN:**\n* **Training GAN needs to reach Nash equilibrium, sometimes it can be done by gradient descent method, sometimes it can't be done. We have not found a good way to reach Nash Equilibrium, so training GAN is unstable compared to VAE or PixelRNN. But I think in practice it is still more stable than training the Boltzmann machine.**\n* **GAN is not suitable for processing discrete forms of data, such as text**\n* **GAN has problems with unstable training, gradient disappearance, and mode collapse (currently resolved)**\n\nGenerally, when the GAN training is unstable, the result is very poor, but it cannot be improved even after the training time is lengthened.\n\n**6.Why is the optimizer in GAN not commonly used for SGD?**\n\n* 1. SGD is easy to oscillate, and it is easy to make GAN training unstable.\n* 2. The purpose of GAN is to find the Nash equilibrium point in the high-dimensional non-convex parameter space. The Nash equilibrium point of GAN is a saddle point, but SGD will only find the local minimum value, because SGD solves the problem of finding the minimum value, GAN It is a game problem.\n\n**7.Why GAN is not suitable for processing text data**\n\n1. Text data is discrete compared to image data, because for text, it is usually necessary to map a word to a high-dimensional vector, and the final predicted output is a one-hot vector, assuming that the output of softmax is ( 0.2, 0.3, 0.1, 0.2, 0.15, 0.05) then become onehot is (0,1,0,0,0,0), if the softmax output is (0.2, 0.25, 0.2, 0.1, 0.15, 0.1), one -hot is still (0, 1, 0, 0, 0, 0), so for the generator, G outputs different results but D gives the same result, and the gradient update information is not very good. Passed to G, so the judgment of the final output of D is meaningless.\n2. In addition, the loss function of GAN is JS divergence, and JS divergence is not suitable for measuring the distance between the distributions that do not want to intersect.\n\n(WGAN uses the wassertein distance instead of the JS divergence, but the ability to generate text is still limited. GAN uses seq-GAN in the generated text, and the product of reinforcement learning)\n\n**8.Some tips for training GAN**\n1. The input is normalized to (-1,1), and the activation function of the last layer uses tanh (except BEGAN)\n2. Using the loss function of wassertein GAN,\n3. If there is tag data, try to use tags. It is also suggested that using reverse tags works well. In addition, tag smoothing, single-sided label smoothing or bilateral label smoothing is used.\n4. Use mini-batch norm, if you don't use batch norm you can use instance norm or weight norm\n5. Avoid using the RELU and pooling layers to reduce the possibility of sparse gradients. You can use the leanrelu activation function.\n6. The optimizer should choose ADAM as much as possible. The learning rate should not be set too large. The initial 1e-4 can be used for reference. In addition, the learning rate can be continuously reduced as the training progresses.\n7. Adding Gaussian noise to the network layer of D is equivalent to a regular"},{"metadata":{"_uuid":"4bfa5d45c232b7eae2278abf66db8729e20cc566"},"cell_type":"markdown","source":"### 1.Load data"},{"metadata":{"trusted":true,"_uuid":"42157d2e35a24ff7a4c413d15ae2bd4dda8859fa"},"cell_type":"code","source":"import tensorflow as tf\nimport numpy as np\nimport matplotlib.pyplot as plt\n\ntf.reset_default_graph()\ntf.set_random_seed(1)\nnp.random.seed(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"73a74b40dc0212dfa346ccd9c2214991f7b7bd00"},"cell_type":"markdown","source":"### 2.Plot the data"},{"metadata":{"trusted":true,"_uuid":"56b59017cdc26d7ef06af624a07db0a1c8fd9c55"},"cell_type":"code","source":"# Hyper Parameters\nBATCH_SIZE = 64\nLR_G = 0.0001 # learning rate for generator\nLR_D = 0.0001 # learning rate for discriminator\nN_IDEAS = 5 # think of this as number of ideas for generating an art work (Generator)\nART_COMPONENTS = 15 # it could be total point G can draw in the canvas\nPAINT_POINTS = np.vstack([np.linspace(-1, 1, ART_COMPONENTS) for _ in range(BATCH_SIZE)])\n\n# show our beautiful painting range\nplt.figure(figsize=(20,8))\nplt.plot(PAINT_POINTS[0], 2 * np.power(PAINT_POINTS[0], 2) + 1, c='#74BCFF', lw=3, label='upper bound')\nplt.plot(PAINT_POINTS[0], 1 * np.power(PAINT_POINTS[0], 2) + 0, c='#FF9359', lw=3, label='lower bound')\nplt.legend(loc='upper right')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b9e9561b3233e17ecb956ca1250ae22cde8c8f6"},"cell_type":"markdown","source":"### 3.Design Art work"},{"metadata":{"trusted":true,"_uuid":"7b861393c369c966dc49f68a355beb96c9cfa690"},"cell_type":"code","source":"def artist_works(): # painting from the famous artist (real target)\n a = np.random.uniform(1, 2, size=BATCH_SIZE)[:, np.newaxis]\n paintings = a * np.power(PAINT_POINTS, 2) + (a-1)\n return paintings","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa2a9355e66019d35fed37744ed5c4cdae5ffbea"},"cell_type":"markdown","source":"### 4.Design of GAN"},{"metadata":{"trusted":true,"_uuid":"afe40242839d4aa3dbb30c86be5b4de4907bde83","_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"with tf.variable_scope('Generator'):\n G_in = tf.placeholder(tf.float32, [None, N_IDEAS]) # random ideas (could from normal distribution)\n G_l1 = tf.layers.dense(G_in, 128, tf.nn.relu)\n G_out = tf.layers.dense(G_l1, ART_COMPONENTS) # making a painting from these random ideas\n\nwith tf.variable_scope('Discriminator'):\n real_art = tf.placeholder(tf.float32, [None, ART_COMPONENTS], name='real_in') # receive art work from the famous artist\n D_l0 = tf.layers.dense(real_art, 128, tf.nn.relu, name='l')\n prob_artist0 = tf.layers.dense(D_l0, 1, tf.nn.sigmoid, name='out') # probability that the art work is made by artist\n # reuse layers for generator\n D_l1 = tf.layers.dense(G_out, 128, tf.nn.relu, name='l', reuse=True) # receive art work from a newbie like G\n prob_artist1 = tf.layers.dense(D_l1, 1, tf.nn.sigmoid, name='out', reuse=True) # probability that the art work is made by artist\n\nD_loss = -tf.reduce_mean(tf.log(prob_artist0) + tf.log(1-prob_artist1))\nG_loss = tf.reduce_mean(tf.log(1-prob_artist1))\n\ntrain_D = tf.train.AdamOptimizer(LR_D).minimize(\n D_loss, var_list=tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope='Discriminator'))\ntrain_G = tf.train.AdamOptimizer(LR_G).minimize(\n G_loss, var_list=tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope='Generator'))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1534ca937f6f665d48d1e98d337fff5cf3a7de74"},"cell_type":"markdown","source":"### 5.Model Running"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"9f0ab3a76ab4cae7e41b5de716d869fe9f9f7c88"},"cell_type":"code","source":"sess = tf.Session()\nsess.run(tf.global_variables_initializer())\n\nplt.ion() # something about continuous plotting\nfor step in range(5000):\n \n artist_paintings = artist_works() # real painting from artist\n G_ideas = np.random.randn(BATCH_SIZE, N_IDEAS)\n G_paintings, pa0, Dl = sess.run([G_out, prob_artist0, D_loss, train_D, train_G], # train and get results\n {G_in: G_ideas, real_art: artist_paintings})[:3]\n\n if step % 50 == 0: # plotting\n plt.figure(figsize=(20,6))\n plt.cla()\n plt.plot(PAINT_POINTS[0], G_paintings[0], c='#4AD631', lw=3, label='Generated painting',)\n plt.plot(PAINT_POINTS[0], 2 * np.power(PAINT_POINTS[0], 2) + 1, c='#74BCFF', lw=3, label='upper bound')\n plt.plot(PAINT_POINTS[0], 1 * np.power(PAINT_POINTS[0], 2) + 0, c='#FF9359', lw=3, label='lower bound')\n plt.text(-.5, 2.3, 'D accuracy=%.2f (0.5 for D to converge)' % pa0.mean(), fontdict={'size': 15})\n plt.text(-.5, 2, 'D score= %.2f (-1.38 for G to converge)' % -Dl, fontdict={'size': 15})\n plt.ylim((0, 3)); plt.legend(loc='upper right', fontsize=12); plt.draw(); plt.pause(0.01)\n\nplt.ioff()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5a053d0bba00cfa14f68e0aba792bd86c436d826"},"cell_type":"markdown","source":"### Notebook is continue updated"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"_uuid":"8164a0499d93e536c1ebba8a4ddcc260a2f23a3a"},"cell_type":"code","source":"# # References :\n\n# * **https://data-flair.training/blogs/tensorflow-applications/**\n# * **https://github.com/MorvanZhou/Tensorflow-Tutorial**\n# * **https://www.guru99.com/tensor-tensorflow.html**","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}