TIP: If filter height and width have the same value, you can instead specify a\nsingle integer for kernel_size—e.g., kernel_size=5.
[batch_size, 28, 28, 32]: the same height and width\ndimensions as the input, but now with 32 channels holding the output from each\nof the filters."},{"metadata":{"_uuid":"43acd27e0ec83b055a0395aa89446c942f7151ef"},"cell_type":"markdown","source":"### Pooling Layer #1\n\nNext, we connect our first pooling layer to the convolutional layer we just\ncreated. We can use the `max_pooling2d()` method in `layers` to construct a\nlayer that performs max pooling with a 2x2 filter and stride of 2:\n\n```\npool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)\n```\n\nAgain, `inputs` specifies the input tensor, with a shape of\n[batch_size, image_height, image_width,\nchannels]. Here, our input tensor is `conv1`, the output from\nthe first convolutional layer, which has a shape of [batch_size,\n28, 28, 32].\n\nNote: As with conv2d(), max_pooling2d() will instead\naccept a shape of [batch_size, channels, \nimage_height, image_width] when passed the argument\ndata_format=channels_first.\n\nThe `pool_size` argument specifies the size of the max pooling filter as\n[height, width] (here, `[2, 2]`). If both\ndimensions have the same value, you can instead specify a single integer (e.g.,\n`pool_size=2`).\n\nThe `strides` argument specifies the size of the stride. Here, we set a stride\nof 2, which indicates that the subregions extracted by the filter should be\nseparated by 2 pixels in both the height and width dimensions (for a 2x2 filter,\nthis means that none of the regions extracted will overlap). If you want to set\ndifferent stride values for height and width, you can instead specify a tuple or\nlist (e.g., `stride=[3, 6]`).\n\nOur output tensor produced by `max_pooling2d()` (`pool1`) has a shape of\n[batch_size, 14, 14, 32]: the 2x2 filter reduces height and width by 50% each."},{"metadata":{"_uuid":"14e082169eee305d533809181d7b81375197f3c2"},"cell_type":"markdown","source":"### Convolutional Layer #2 and Pooling Layer #2\n\nWe can connect a second convolutional and pooling layer to our CNN using\n`conv2d()` and `max_pooling2d()` as before. For convolutional layer #2, we\nconfigure 64 5x5 filters with ReLU activation, and for pooling layer #2, we use\nthe same specs as pooling layer #1 (a 2x2 max pooling filter with stride of 2):\n\n```\nconv2 = tf.layers.conv2d(\n inputs=pool1,\n filters=64,\n kernel_size=[5, 5],\n padding=\"same\",\n activation=tf.nn.relu)\n\npool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)\n```\n\nNote that convolutional layer #2 takes the output tensor of our first pooling\nlayer (`pool1`) as input, and produces the tensor `conv2` as output. `conv2`\nhas a shape of [batch_size, 14, 14, 64], the same height and width as `pool1` (due to `padding=\"same\"`), and 64 channels for the 64\nfilters applied.\n\nPooling layer #2 takes `conv2` as input, producing `pool2` as output. `pool2`\nhas shape [batch_size, 7, 7, 64] (50% reduction of height and width from `conv2`)."},{"metadata":{"_uuid":"3090622c0a72d5858e4a2adc2a103f9b9dbaebec"},"cell_type":"markdown","source":"### Dense Layer\n\nNext, we want to add a dense layer (with 1,024 neurons and ReLU activation) to\nour CNN to perform classification on the features extracted by the\nconvolution/pooling layers. Before we connect the layer, however, we'll flatten\nour feature map (`pool2`) to shape [batch_size,\nfeatures], so that our tensor has only two dimensions:\n\n```\npool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])\n```\n\nIn the `reshape()` operation above, the `-1` signifies that the *`batch_size`*\ndimension will be dynamically calculated based on the number of examples in our\ninput data. Each example has 7 (`pool2` height) * 7 (`pool2` width) * 64\n(`pool2` channels) features, so we want the `features` dimension to have a value\nof 7 * 7 * 64 (3136 in total). The output tensor, `pool2_flat`, has shape\n[batch_size, 3136].\n\nNow, we can use the `dense()` method in `layers` to connect our dense layer as\nfollows:\n\n```\ndense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)\n```\n\nThe `inputs` argument specifies the input tensor: our flattened feature map,\n`pool2_flat`. The `units` argument specifies the number of neurons in the dense\nlayer (1,024). The `activation` argument takes the activation function; again,\nwe'll use `tf.nn.relu` to add ReLU activation.\n\nTo help improve the results of our model, we also apply dropout regularization\nto our dense layer, using the `dropout` method in `layers`:\n\n```\ndropout = tf.layers.dropout(\n inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN)\n```\n\nAgain, `inputs` specifies the input tensor, which is the output tensor from our\ndense layer (`dense`).\n\nThe `rate` argument specifies the dropout rate; here, we use `0.4`, which means\n40% of the elements will be randomly dropped out during training.\n\nThe `training` argument takes a boolean specifying whether or not the model is\ncurrently being run in training mode; dropout will only be performed if\n`training` is `True`. Here, we check if the `mode` passed to our model function\n`cnn_model_fn` is `TRAIN` mode.\n\nOur output tensor `dropout` has shape [batch_size, 1024]."},{"metadata":{"_uuid":"615daa1eb0b5f98158cc66b43f60321cf67fdd41"},"cell_type":"markdown","source":"### Logits Layer\n\nThe final layer in our neural network is the logits layer, which will return the\nraw values for our predictions. We create a dense layer with 10 neurons (one for\neach target class 0–9), with linear activation (the default):\n\n```\nlogits = tf.layers.dense(inputs=dropout, units=10)\n```\n\nOur final output tensor of the CNN, `logits`, has shape `[batch_size, 10]`."},{"metadata":{"_uuid":"417b3cf0d61f79334bb52bca9736cc1798ddb5b7"},"cell_type":"markdown","source":"### Generate Predictions {#generate_predictions}\n\nThe logits layer of our model returns our predictions as raw values in a\n[batch_size, 10]-dimensional tensor. Let's convert these\nraw values into two different formats that our model function can return:\n\n* The **predicted class** for each example: a digit from 0–9.\n* The **probabilities** for each possible target class for each example: the\n probability that the example is a 0, is a 1, is a 2, etc.\n\nFor a given example, our predicted class is the element in the corresponding row\nof the logits tensor with the highest raw value. We can find the index of this\nelement using the `tf.argmax`\nfunction:\n\n```\ntf.argmax(input=logits, axis=1)\n```\n\nThe `input` argument specifies the tensor from which to extract maximum\nvalues—here `logits`. The `axis` argument specifies the axis of the `input`\ntensor along which to find the greatest value. Here, we want to find the largest\nvalue along the dimension with index of 1, which corresponds to our predictions\n(recall that our logits tensor has shape [batch_size,\n10]).\n\nWe can derive probabilities from our logits layer by applying softmax activation\nusing `tf.nn.softmax`:\n\n```\ntf.nn.softmax(logits, name=\"softmax_tensor\")\n```\n\nNote: We use the `name` argument to explicitly name this operation `softmax_tensor`, so we can reference it later. (We'll set up logging for the softmax values in [\"Set Up a Logging Hook\"](#set-up-a-logging-hook)).\n\nWe compile our predictions in a dict, and return an `EstimatorSpec` object:\n\n```\npredictions = {\n \"classes\": tf.argmax(input=logits, axis=1),\n \"probabilities\": tf.nn.softmax(logits, name=\"softmax_tensor\")\n}\nif mode == tf.estimator.ModeKeys.PREDICT:\n return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)\n```"},{"metadata":{"_uuid":"5495dde88ab5f50c4ddf971120c8659dda2b441f"},"cell_type":"markdown","source":"### Calculate Loss {#calculating-loss}\n\nFor both training and evaluation, we need to define a\n[loss function](https://en.wikipedia.org/wiki/Loss_function)\nthat measures how closely the model's predictions match the target classes. For\nmulticlass classification problems like MNIST,\n[cross entropy](https://en.wikipedia.org/wiki/Cross_entropy) is typically used\nas the loss metric. The following code calculates cross entropy when the model\nruns in either `TRAIN` or `EVAL` mode:\n\n```\nloss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)\n```\n\nLet's take a closer look at what's happening above.\n\nOur `labels` tensor contains a list of prediction indices for our examples, e.g. `[1,\n9, ...]`. `logits` contains the linear outputs of our last layer. \n\n`tf.losses.sparse_softmax_cross_entropy`, calculates the softmax crossentropy\n(aka: categorical crossentropy, negative log-likelihood) from these two inputs\nin an efficient, numerically stable way."},{"metadata":{"_uuid":"05b38836e1ecf819620c2b4ef4a57201d0e17d76"},"cell_type":"markdown","source":"### Configure the Training Op\n\nIn the previous section, we defined loss for our CNN as the softmax\ncross-entropy of the logits layer and our labels. Let's configure our model to\noptimize this loss value during training. We'll use a learning rate of 0.001 and\n[stochastic gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent)\nas the optimization algorithm:\n\n```\nif mode == tf.estimator.ModeKeys.TRAIN:\n optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)\n train_op = optimizer.minimize(\n loss=loss,\n global_step=tf.train.get_global_step())\n return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)\n```"},{"metadata":{"_uuid":"7216fe51880596bdfe9d2475d7ff8bcb22caa613"},"cell_type":"markdown","source":"### Add evaluation metrics\n\nTo add accuracy metric in our model, we define `eval_metric_ops` dict in EVAL\nmode as follows:\n\n```\neval_metric_ops = {\n \"accuracy\": tf.metrics.accuracy(\n labels=labels, predictions=predictions[\"classes\"])\n}\nreturn tf.estimator.EstimatorSpec(\n mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)\n```"},{"metadata":{"_uuid":"d5d8a29662767394ef20bc50eb85af211e3a51f0"},"cell_type":"markdown","source":"\n## Training and Evaluating the CNN MNIST Classifier\n\nWe've coded our MNIST CNN model function; now we're ready to train and evaluate\nit.\n\n### Load Training and Test Data\n\nFirst, let's load our training and test data with the following code:"},{"metadata":{"trusted":true,"_uuid":"befd62f26d3db4ff1c50fa04e9e65d703d16a518"},"cell_type":"code","source":"# Load training and eval data\n((train_data, train_labels),\n (eval_data, eval_labels)) = tf.keras.datasets.mnist.load_data()\n\ntrain_data = train_data/np.float32(255)\ntrain_labels = train_labels.astype(np.int32) # not required\n\neval_data = eval_data/np.float32(255)\neval_labels = eval_labels.astype(np.int32) # not required","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"25335bd223a697ab3a183d074f4853889f8db71d"},"cell_type":"markdown","source":"We store the training feature data (the raw pixel values for 55,000 images of\nhand-drawn digits) and training labels (the corresponding value from 0–9 for\neach image) as [numpy\narrays](https://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html)\nin `train_data` and `train_labels`, respectively. Similarly, we store the\nevaluation feature data (10,000 images) and evaluation labels in `eval_data`\nand `eval_labels`, respectively\n\n### Create the Estimator {#create-the-estimator}\n\nNext, let's create an `Estimator` (a TensorFlow class for performing high-level\nmodel training, evaluation, and inference) for our model. Add the following code\nto `main()`:"},{"metadata":{"trusted":true,"_uuid":"8e398c758ac9dd24c1a1a764313a92e09e6e2a57"},"cell_type":"code","source":"# Create the Estimator\nmnist_classifier = tf.estimator.Estimator(\n model_fn=cnn_model_fn, model_dir=\"/tmp/mnist_convnet_model\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"af225e8b4d1b0314f9d150ddc516949fe4351468"},"cell_type":"markdown","source":"The `model_fn` argument specifies the model function to use for training,\nevaluation, and prediction; we pass it the `cnn_model_fn` we created in\n[\"Building the CNN MNIST Classifier.\"](#building-the-cnn-mnist-classifier) The\n`model_dir` argument specifies the directory where model data (checkpoints) will\nbe saved (here, we specify the temp directory `/tmp/mnist_convnet_model`, but\nfeel free to change to another directory of your choice).\n\nNote: For an in-depth walkthrough of the TensorFlow `Estimator` API, see the tutorial [Creating Estimators in tf.estimator](../../guide/custom_estimators.md)."},{"metadata":{"_uuid":"a57865907bd73a7e05a65c2c0fa2ef001e9f32ae"},"cell_type":"markdown","source":"### Set Up a Logging Hook {#set_up_a_logging_hook}\n\nSince CNNs can take a while to train, let's set up some logging so we can track\nprogress during training. We can use TensorFlow's `tf.train.SessionRunHook` to create a\n`tf.train.LoggingTensorHook`\nthat will log the probability values from the softmax layer of our CNN. Add the\nfollowing to `main()`:"},{"metadata":{"trusted":true,"_uuid":"8f0789f87aed15da79091d3b6136bac5e1275dc3"},"cell_type":"code","source":"# Set up logging for predictions\ntensors_to_log = {\"probabilities\": \"softmax_tensor\"}\n\nlogging_hook = tf.train.LoggingTensorHook(\n tensors=tensors_to_log, every_n_iter=50)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"02e63477de9d5b2681fbb39047ebb28126f1ccee"},"cell_type":"markdown","source":"We store a dict of the tensors we want to log in `tensors_to_log`. Each key is a\nlabel of our choice that will be printed in the log output, and the\ncorresponding label is the name of a `Tensor` in the TensorFlow graph. Here, our\n`probabilities` can be found in `softmax_tensor`, the name we gave our softmax\noperation earlier when we generated the probabilities in `cnn_model_fn`.\n\nNote: If you don't explicitly assign a name to an operation via the `name` argument, TensorFlow will assign a default name. A couple easy ways to discover the names applied to operations are to visualize your graph on [TensorBoard](../../guide/graph_viz.md)) or to enable the [TensorFlow Debugger (tfdbg)](../../guide/debugger.md).\n\nNext, we create the `LoggingTensorHook`, passing `tensors_to_log` to the\n`tensors` argument. We set `every_n_iter=50`, which specifies that probabilities\nshould be logged after every 50 steps of training.\n\n### Train the Model\n\nNow we're ready to train our model, which we can do by creating `train_input_fn`\nand calling `train()` on `mnist_classifier`. In the `numpy_input_fn` call, we pass the training feature data and labels to\n`x` (as a dict) and `y`, respectively. We set a `batch_size` of `100` (which\nmeans that the model will train on minibatches of 100 examples at each step).\n`num_epochs=None` means that the model will train until the specified number of\nsteps is reached. We also set `shuffle=True` to shuffle the training data. Then train the model a single step and log the output:"},{"metadata":{"trusted":true,"_uuid":"13e16c82c698d1a209fdac5c8f2c2fe8b3f26b74"},"cell_type":"code","source":"# Train the model\ntrain_input_fn = tf.estimator.inputs.numpy_input_fn(\n x={\"x\": train_data},\n y=train_labels,\n batch_size=100,\n num_epochs=None,\n shuffle=True)\n\n# train one step and display the probabilties\nmnist_classifier.train(\n input_fn=train_input_fn,\n steps=1,\n hooks=[logging_hook])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b7819e18bbc2af228bcfdab9f5f6e0d764e1b921"},"cell_type":"markdown","source":"Now—without logging each step—set `steps=1000` to train the model longer, but in a reasonable time to run this example. Training CNNs is computationally intensive. To increase the accuracy of your model, increase the number of `steps` passed to `train()`, like 20,000 steps. "},{"metadata":{"trusted":true,"_uuid":"c2673de4a2cceced373049a64b6822b078da3260"},"cell_type":"code","source":"mnist_classifier.train(input_fn=train_input_fn, steps=1000)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"38cd2a5baef08bae7f0742e679851bb9c3b5b17f"},"cell_type":"markdown","source":"### Evaluate the Model\n\nOnce training is complete, we want to evaluate our model to determine its\naccuracy on the MNIST test set. We call the `evaluate` method, which evaluates\nthe metrics we specified in `eval_metric_ops` argument in the `model_fn`.\nAdd the following to `main()`:"},{"metadata":{"trusted":true,"_uuid":"4f1aac70ced7d3cdb36bf9a8897cd3e6d63c0b29"},"cell_type":"code","source":"eval_input_fn = tf.estimator.inputs.numpy_input_fn(\n x={\"x\": eval_data},\n y=eval_labels,\n num_epochs=1,\n shuffle=False)\n\neval_results = mnist_classifier.evaluate(input_fn=eval_input_fn)\nprint(eval_results)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"14583fab78b9f7d61c97efc0179d7545f6f98261"},"cell_type":"markdown","source":"To create `eval_input_fn`, we set `num_epochs=1`, so that the model evaluates\nthe metrics over one epoch of data and returns the result. We also set\n`shuffle=False` to iterate through the data sequentially.\n\n## Additional Resources\n\nTo learn more about TensorFlow Estimators and CNNs in TensorFlow, see the\nfollowing resources:\n\n* [Creating Estimators in tf.estimator](../../guide/custom_estimators.md)\n provides an introduction to the TensorFlow Estimator API. It walks through\n configuring an Estimator, writing a model function, calculating loss, and\n defining a training op.\n* [Advanced Convolutional Neural Networks](../../tutorials/images/deep_cnn.md) walks through how to build a MNIST CNN classification model\n *without estimators* using lower-level TensorFlow operations."},{"metadata":{"_uuid":"f08e53b54475ec478e6b0259118b6af08cd6dde5"},"cell_type":"markdown","source":"## License\n\n##### Copyright 2018 The TensorFlow Authors."},{"metadata":{"trusted":true,"_uuid":"c5f7b20a6017184bd3f4f53a92216e50177b6f9f"},"cell_type":"code","source":"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# https://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"14484f13aeb7702dd785ba1924980b3807b8e4e7"},"cell_type":"markdown","source":"## Deep Learning Basics with TensorFlow\n\nThis tutorial accompanies the [lecture on Deep Learning Basics](https://www.youtube.com/watch?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf&v=O5xeyoRL95U) given as part of [MIT Deep Learning](https://deeplearning.mit.edu). Acknowledgement to amazing people involved is provided throughout the tutorial and at the end. You can watch the video on YouTube:\n\nIn this tutorial, we mention seven important types/concepts/approaches in deep learning, introducing the first 2 and providing pointers to tutorials on the others. Here is a visual representation of the seven:\n\n\n\nAt a high-level, neural networks are either encoders, decoders, or a combination of both. Encoders find patterns in raw data to form compact, useful representations. Decoders generate new data or high-resolution useful infomation from those representations. As the lecture describes, deep learning discovers ways to **represent** the world so that we can reason about it. The rest is clever methods that help use deal effectively with visual information, language, sound (#1-6) and even act in a world based on this information and occasional rewards (#7).\n\n1. **Feed Forward Neural Networks (FFNNs)** - classification and regression based on features. See [Part 1](#Part-1:-Boston-Housing-Price-Prediction-with-Feed-Forward-Neural-Networks) of this tutorial for an example.\n2. **Convolutional Neural Networks (CNNs)** - image classification, object detection, video action recognition, etc. See [Part 2](#Part-2:-Classification-of-MNIST-Dreams-with-Convolution-Neural-Networks) of this tutorial for an example.\n3. **Recurrent Neural Networks (RNNs)** - language modeling, speech recognition/generation, etc. See [this TF tutorial on text generation](https://www.tensorflow.org/tutorials/sequences/text_generation) for an example.\n4. **Encoder Decoder Architectures** - semantic segmentation, machine translation, etc. See [our tutorial on semantic segmentation](https://github.com/lexfridman/mit-deep-learning/blob/master/tutorial_driving_scene_segmentation/tutorial_driving_scene_segmentation.ipynb) for an example.\n5. **Autoencoder** - unsupervised embeddings, denoising, etc.\n6. **Generative Adversarial Networks (GANs)** - unsupervised generation of realistic images, etc. See [this TF tutorial on DCGANs](https://github.com/tensorflow/tensorflow/blob/r1.11/tensorflow/contrib/eager/python/examples/generative_examples/dcgan.ipynb) for an example.\n7. **Deep Reinforcement Learning** - game playing, robotics in simulation, self-play, neural arhitecture search, etc. We'll be releasing notebooks on this soon and will link them here.\n\n## License\n\nMIT License\n\nCopyright (c) 2019 Lex Fridman\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n\n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Awesome Deep Learning Resources\n\n## Awesome Deep Learning - Part I\n\n### Table of Content - Part I\n\n* **[Free Online Books](#free-online-books)** \n\n* **[Courses](#courses)** \n\n* **[Videos and Lectures](#videos-and-lectures)** \n\n* **[Papers](#papers)** \n\n* **[Tutorials](#tutorials)** \n\n* **[Researchers](#researchers)** \n\n* **[Websites](#websites)** \n\n* **[Datasets](#datasets)**\n\n* **[Conferences](#Conferences)**\n\n* **[Frameworks](#frameworks)** \n\n* **[Tools](#tools)** \n\n* **[Miscellaneous](#miscellaneous)** \n\n \n### Free Online Books\n\n1. [Deep Learning](http://www.deeplearningbook.org/) by Yoshua Bengio, Ian Goodfellow and Aaron Courville (05/07/2015)\n2. [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/) by Michael Nielsen (Dec 2014)\n3. [Deep Learning](http://research.microsoft.com/pubs/209355/DeepLearning-NowPublishing-Vol7-SIG-039.pdf) by Microsoft Research (2013) \n4. [Deep Learning Tutorial](http://deeplearning.net/tutorial/deeplearning.pdf) by LISA lab, University of Montreal (Jan 6 2015)\n5. [neuraltalk](https://github.com/karpathy/neuraltalk) by Andrej Karpathy : numpy-based RNN/LSTM implementation\n6. [An introduction to genetic algorithms](https://svn-d1.mpi-inf.mpg.de/AG1/MultiCoreLab/papers/ebook-fuzzy-mitchell-99.pdf)\n7. [Artificial Intelligence: A Modern Approach](http://aima.cs.berkeley.edu/)\n8. [Deep Learning in Neural Networks: An Overview](http://arxiv.org/pdf/1404.7828v4.pdf)\n9. [Artificial intelligence and machine learning: Topic wise explanation](https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/)\n \n### Courses\n\n1. [Machine Learning - Stanford](https://class.coursera.org/ml-005) by Andrew Ng in Coursera (2010-2014)\n2. [Machine Learning - Caltech](http://work.caltech.edu/lectures.html) by Yaser Abu-Mostafa (2012-2014)\n3. [Machine Learning - Carnegie Mellon](http://www.cs.cmu.edu/~tom/10701_sp11/lectures.shtml) by Tom Mitchell (Spring 2011)\n2. [Neural Networks for Machine Learning](https://class.coursera.org/neuralnets-2012-001) by Geoffrey Hinton in Coursera (2012)\n3. [Neural networks class](https://www.youtube.com/playlist?list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH) by Hugo Larochelle from Université de Sherbrooke (2013)\n4. [Deep Learning Course](http://cilvr.cs.nyu.edu/doku.php?id=deeplearning:slides:start) by CILVR lab @ NYU (2014)\n5. [A.I - Berkeley](https://courses.edx.org/courses/BerkeleyX/CS188x_1/1T2013/courseware/) by Dan Klein and Pieter Abbeel (2013)\n6. [A.I - MIT](http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-034-artificial-intelligence-fall-2010/lecture-videos/) by Patrick Henry Winston (2010)\n7. [Vision and learning - computers and brains](http://web.mit.edu/course/other/i2course/www/vision_and_learning_fall_2013.html) by Shimon Ullman, Tomaso Poggio, Ethan Meyers @ MIT (2013)\n9. [Convolutional Neural Networks for Visual Recognition - Stanford](http://vision.stanford.edu/teaching/cs231n/syllabus.html) by Fei-Fei Li, Andrej Karpathy (2017)\n10. [Deep Learning for Natural Language Processing - Stanford](http://cs224d.stanford.edu/)\n11. [Neural Networks - usherbrooke](http://info.usherbrooke.ca/hlarochelle/neural_networks/content.html)\n12. [Machine Learning - Oxford](https://www.cs.ox.ac.uk/people/nando.defreitas/machinelearning/) (2014-2015)\n13. [Deep Learning - Nvidia](https://developer.nvidia.com/deep-learning-courses) (2015)\n14. [Graduate Summer School: Deep Learning, Feature Learning](https://www.youtube.com/playlist?list=PLHyI3Fbmv0SdzMHAy0aN59oYnLy5vyyTA) by Geoffrey Hinton, Yoshua Bengio, Yann LeCun, Andrew Ng, Nando de Freitas and several others @ IPAM, UCLA (2012)\n15. [Deep Learning - Udacity/Google](https://www.udacity.com/course/deep-learning--ud730) by Vincent Vanhoucke and Arpan Chakraborty (2016)\n16. [Deep Learning - UWaterloo](https://www.youtube.com/playlist?list=PLehuLRPyt1Hyi78UOkMPWCGRxGcA9NVOE) by Prof. Ali Ghodsi at University of Waterloo (2015)\n17. [Statistical Machine Learning - CMU](https://www.youtube.com/watch?v=azaLcvuql_g&list=PLjbUi5mgii6BWEUZf7He6nowWvGne_Y8r) by Prof. Larry Wasserman\n18. [Deep Learning Course](https://www.college-de-france.fr/site/en-yann-lecun/course-2015-2016.htm) by Yann LeCun (2016)\n19. [Designing, Visualizing and Understanding Deep Neural Networks-UC Berkeley](https://www.youtube.com/playlist?list=PLkFD6_40KJIxopmdJF_CLNqG3QuDFHQUm)\n20. [UVA Deep Learning Course](http://uvadlc.github.io) MSc in Artificial Intelligence for the University of Amsterdam.\n21. [MIT 6.S094: Deep Learning for Self-Driving Cars](http://selfdrivingcars.mit.edu/)\n22. [MIT 6.S191: Introduction to Deep Learning](http://introtodeeplearning.com/)\n23. [Berkeley CS 294: Deep Reinforcement Learning](http://rll.berkeley.edu/deeprlcourse/)\n24. [Keras in Motion video course](https://www.manning.com/livevideo/keras-in-motion)\n25. [Practical Deep Learning For Coders](http://course.fast.ai/) by Jeremy Howard - Fast.ai\n26. [Introduction to Deep Learning](http://deeplearning.cs.cmu.edu/) by Prof. Bhiksha Raj (2017)\n27. [Google Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course/)\n\n### Videos and Lectures\n\n1. [How To Create A Mind](https://www.youtube.com/watch?v=RIkxVci-R4k) By Ray Kurzweil\n2. [Deep Learning, Self-Taught Learning and Unsupervised Feature Learning](https://www.youtube.com/watch?v=n1ViNeWhC24) By Andrew Ng\n3. [Recent Developments in Deep Learning](https://www.youtube.com/watch?v=vShMxxqtDDs&index=3&list=PL78U8qQHXgrhP9aZraxTT5-X1RccTcUYT) By Geoff Hinton\n4. [The Unreasonable Effectiveness of Deep Learning](https://www.youtube.com/watch?v=sc-KbuZqGkI) by Yann LeCun\n5. [Deep Learning of Representations](https://www.youtube.com/watch?v=4xsVFLnHC_0) by Yoshua bengio\n6. [Principles of Hierarchical Temporal Memory](https://www.youtube.com/watch?v=6ufPpZDmPKA) by Jeff Hawkins\n7. [Machine Learning Discussion Group - Deep Learning w/ Stanford AI Lab](https://www.youtube.com/watch?v=2QJi0ArLq7s&list=PL78U8qQHXgrhP9aZraxTT5-X1RccTcUYT) by Adam Coates\n8. [Making Sense of the World with Deep Learning](http://vimeo.com/80821560) By Adam Coates \n9. [Demystifying Unsupervised Feature Learning ](https://www.youtube.com/watch?v=wZfVBwOO0-k) By Adam Coates \n10. [Visual Perception with Deep Learning](https://www.youtube.com/watch?v=3boKlkPBckA) By Yann LeCun\n11. [The Next Generation of Neural Networks](https://www.youtube.com/watch?v=AyzOUbkUf3M) By Geoffrey Hinton at GoogleTechTalks\n12. [The wonderful and terrifying implications of computers that can learn](http://www.ted.com/talks/jeremy_howard_the_wonderful_and_terrifying_implications_of_computers_that_can_learn) By Jeremy Howard at TEDxBrussels\n13. [Unsupervised Deep Learning - Stanford](http://web.stanford.edu/class/cs294a/handouts.html) by Andrew Ng in Stanford (2011)\n14. [Natural Language Processing](http://web.stanford.edu/class/cs224n/handouts/) By Chris Manning in Stanford\n15. [A beginners Guide to Deep Neural Networks](http://googleresearch.blogspot.com/2015/09/a-beginners-guide-to-deep-neural.html) By Natalie Hammel and Lorraine Yurshansky\n16. [Deep Learning: Intelligence from Big Data](https://www.youtube.com/watch?v=czLI3oLDe8M) by Steve Jurvetson (and panel) at VLAB in Stanford. \n17. [Introduction to Artificial Neural Networks and Deep Learning](https://www.youtube.com/watch?v=FoO8qDB8gUU) by Leo Isikdogan at Motorola Mobility HQ\n18. [NIPS 2016 lecture and workshop videos](https://nips.cc/Conferences/2016/Schedule) - NIPS 2016\n19. [Deep Learning Crash Course](https://www.youtube.com/watch?v=oS5fz_mHVz0&list=PLWKotBjTDoLj3rXBL-nEIPRN9V3a9Cx07): a series of mini-lectures by Leo Isikdogan on YouTube (2018)\n\n### Papers\n*You can also find the most cited deep learning papers from [here](https://github.com/terryum/awesome-deep-learning-papers)*\n\n1. [ImageNet Classification with Deep Convolutional Neural Networks](http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf)\n2. [Using Very Deep Autoencoders for Content Based Image Retrieval](http://www.cs.toronto.edu/~hinton/absps/esann-deep-final.pdf)\n3. [Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~lisa/pointeurs/TR1312.pdf)\n4. [CMU’s list of papers](http://deeplearning.cs.cmu.edu/)\n5. [Neural Networks for Named Entity Recognition](http://nlp.stanford.edu/~socherr/pa4_ner.pdf) [zip](http://nlp.stanford.edu/~socherr/pa4-ner.zip)\n6. [Training tricks by YB](http://www.iro.umontreal.ca/~bengioy/papers/YB-tricks.pdf)\n7. [Geoff Hinton's reading list (all papers)](http://www.cs.toronto.edu/~hinton/deeprefs.html)\n8. [Supervised Sequence Labelling with Recurrent Neural Networks](http://www.cs.toronto.edu/~graves/preprint.pdf)\n9. [Statistical Language Models based on Neural Networks](http://www.fit.vutbr.cz/~imikolov/rnnlm/thesis.pdf)\n10. [Training Recurrent Neural Networks](http://www.cs.utoronto.ca/~ilya/pubs/ilya_sutskever_phd_thesis.pdf)\n11. [Recursive Deep Learning for Natural Language Processing and Computer Vision](http://nlp.stanford.edu/~socherr/thesis.pdf)\n12. [Bi-directional RNN](http://www.di.ufpe.br/~fnj/RNA/bibliografia/BRNN.pdf)\n13. [LSTM](http://web.eecs.utk.edu/~itamar/courses/ECE-692/Bobby_paper1.pdf)\n14. [GRU - Gated Recurrent Unit](http://arxiv.org/pdf/1406.1078v3.pdf)\n15. [GFRNN](http://arxiv.org/pdf/1502.02367v3.pdf) [.](http://jmlr.org/proceedings/papers/v37/chung15.pdf) [.](http://jmlr.org/proceedings/papers/v37/chung15-supp.pdf)\n16. [LSTM: A Search Space Odyssey](http://arxiv.org/pdf/1503.04069v1.pdf)\n17. [A Critical Review of Recurrent Neural Networks for Sequence Learning](http://arxiv.org/pdf/1506.00019v1.pdf)\n18. [Visualizing and Understanding Recurrent Networks](http://arxiv.org/pdf/1506.02078v1.pdf)\n19. [Wojciech Zaremba, Ilya Sutskever, An Empirical Exploration of Recurrent Network Architectures](http://jmlr.org/proceedings/papers/v37/jozefowicz15.pdf)\n20. [Recurrent Neural Network based Language Model](http://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf)\n21. [Extensions of Recurrent Neural Network Language Model](http://www.fit.vutbr.cz/research/groups/speech/publi/2011/mikolov_icassp2011_5528.pdf)\n22. [Recurrent Neural Network based Language Modeling in Meeting Recognition](http://www.fit.vutbr.cz/~imikolov/rnnlm/ApplicationOfRNNinMeetingRecognition_IS2011.pdf)\n23. [Deep Neural Networks for Acoustic Modeling in Speech Recognition](http://cs224d.stanford.edu/papers/maas_paper.pdf)\n24. [Speech Recognition with Deep Recurrent Neural Networks](http://www.cs.toronto.edu/~fritz/absps/RNN13.pdf)\n25. [Reinforcement Learning Neural Turing Machines](http://arxiv.org/pdf/1505.00521v1)\n26. [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](http://arxiv.org/pdf/1406.1078v3.pdf)\n27. [Google - Sequence to Sequence Learning with Neural Networks](http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf)\n28. [Memory Networks](http://arxiv.org/pdf/1410.3916v10)\n29. [Policy Learning with Continuous Memory States for Partially Observed Robotic Control](http://arxiv.org/pdf/1507.01273v1)\n30. [Microsoft - Jointly Modeling Embedding and Translation to Bridge Video and Language](http://arxiv.org/pdf/1505.01861v1.pdf)\n31. [Neural Turing Machines](http://arxiv.org/pdf/1410.5401v2.pdf)\n32. [Ask Me Anything: Dynamic Memory Networks for Natural Language Processing](http://arxiv.org/pdf/1506.07285v1.pdf)\n33. [Mastering the Game of Go with Deep Neural Networks and Tree Search](http://www.nature.com/nature/journal/v529/n7587/pdf/nature16961.pdf)\n34. [Batch Normalization](https://arxiv.org/abs/1502.03167)\n35. [Residual Learning](https://arxiv.org/pdf/1512.03385v1.pdf)\n36. [Image-to-Image Translation with Conditional Adversarial Networks](https://arxiv.org/pdf/1611.07004v1.pdf) \n37. [Berkeley AI Research (BAIR) Laboratory](https://arxiv.org/pdf/1611.07004v1.pdf) \n38. [MobileNets by Google](https://arxiv.org/abs/1704.04861)\n39. [Cross Audio-Visual Recognition in the Wild Using Deep Learning](https://arxiv.org/abs/1706.05739)\n40. [Dynamic Routing Between Capsules](https://arxiv.org/abs/1710.09829)\n41. [Matrix Capsules With Em Routing](https://openreview.net/pdf?id=HJWLfGWRb)\n42. [Efficient BackProp](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf)\n43. [Collection of Popular Deep Learning Papers](https://github.com/ArunkumarRamanan/Computer-Science-Resources)\n\n### Tutorials\n\n1. [UFLDL Tutorial 1](http://deeplearning.stanford.edu/wiki/index.php/UFLDL_Tutorial)\n2. [UFLDL Tutorial 2](http://ufldl.stanford.edu/tutorial/supervised/LinearRegression/)\n3. [Deep Learning for NLP (without Magic)](http://www.socher.org/index.php/DeepLearningTutorial/DeepLearningTutorial)\n4. [A Deep Learning Tutorial: From Perceptrons to Deep Networks](http://www.toptal.com/machine-learning/an-introduction-to-deep-learning-from-perceptrons-to-deep-networks)\n5. [Deep Learning from the Bottom up](http://www.metacademy.org/roadmaps/rgrosse/deep_learning)\n6. [Theano Tutorial](http://deeplearning.net/tutorial/deeplearning.pdf)\n7. [Neural Networks for Matlab](http://uk.mathworks.com/help/pdf_doc/nnet/nnet_ug.pdf)\n8. [Using convolutional neural nets to detect facial keypoints tutorial](http://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/)\n9. [Torch7 Tutorials](https://github.com/clementfarabet/ipam-tutorials/tree/master/th_tutorials)\n10. [The Best Machine Learning Tutorials On The Web](https://github.com/josephmisiti/machine-learning-module)\n11. [VGG Convolutional Neural Networks Practical](http://www.robots.ox.ac.uk/~vgg/practicals/cnn/index.html)\n12. [TensorFlow tutorials](https://github.com/nlintz/TensorFlow-Tutorials)\n13. [More TensorFlow tutorials](https://github.com/pkmital/tensorflow_tutorials)\n13. [TensorFlow Python Notebooks](https://github.com/aymericdamien/TensorFlow-Examples)\n14. [Keras and Lasagne Deep Learning Tutorials](https://github.com/Vict0rSch/deep_learning)\n15. [Classification on raw time series in TensorFlow with a LSTM RNN](https://github.com/guillaume-chevalier/LSTM-Human-Activity-Recognition)\n16. [Using convolutional neural nets to detect facial keypoints tutorial](http://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/)\n17. [TensorFlow-World](https://github.com/astorfi/TensorFlow-World)\n18. [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python)\n19. [Grokking Deep Learning](https://www.manning.com/books/grokking-deep-learning)\n20. [Deep Learning for Search](https://www.manning.com/books/deep-learning-for-search)\n21. [Keras Tutorial: Content Based Image Retrieval Using a Convolutional Denoising Autoencoder](https://blog.sicara.com/keras-tutorial-content-based-image-retrieval-convolutional-denoising-autoencoder-dc91450cc511)\n22. [Pytorch Tutorial by Yunjey Choi](https://github.com/yunjey/pytorch-tutorial)\n23. [Practical Machine Learning with PyTorch](https://www.kaggle.com/arunkumarramanan/practical-machine-learning-with-pytorch)\n24. [Hands-On Machine Learning with scikit-learn and TensorFlow](https://www.kaggle.com/arunkumarramanan/hands-on-ml-with-scikit-learn-and-tensorflow)\n25. [Awesome TensorFlow Tutorials](https://www.kaggle.com/arunkumarramanan/tensorflow-tutorial-and-examples)\n26. [Awesome TensorFlow Resources](https://www.kaggle.com/arunkumarramanan/awesome-tensorflow-resources)\n\n## Researchers\n\n1. [Aaron Courville](http://aaroncourville.wordpress.com)\n2. [Abdel-rahman Mohamed](http://www.cs.toronto.edu/~asamir/)\n3. [Adam Coates](http://cs.stanford.edu/~acoates/)\n4. [Alex Acero](http://research.microsoft.com/en-us/people/alexac/)\n5. [ Alex Krizhevsky ](http://www.cs.utoronto.ca/~kriz/index.html)\n6. [ Alexander Ilin ](http://users.ics.aalto.fi/alexilin/)\n7. [ Amos Storkey ](http://homepages.inf.ed.ac.uk/amos/)\n8. [ Andrej Karpathy ](http://cs.stanford.edu/~karpathy/)\n9. [ Andrew M. Saxe ](http://www.stanford.edu/~asaxe/)\n10. [ Andrew Ng ](http://www.cs.stanford.edu/people/ang/)\n11. [ Andrew W. Senior ](http://research.google.com/pubs/author37792.html)\n12. [ Andriy Mnih ](http://www.gatsby.ucl.ac.uk/~amnih/)\n13. [ Ayse Naz Erkan ](http://www.cs.nyu.edu/~naz/)\n14. [ Benjamin Schrauwen ](http://reslab.elis.ugent.be/benjamin)\n15. [ Bernardete Ribeiro ](https://www.cisuc.uc.pt/people/show/2020)\n16. [ Bo David Chen ](http://vision.caltech.edu/~bchen3/Site/Bo_David_Chen.html)\n17. [ Boureau Y-Lan ](http://cs.nyu.edu/~ylan/)\n18. [ Brian Kingsbury ](http://researcher.watson.ibm.com/researcher/view.php?person=us-bedk)\n19. [ Christopher Manning ](http://nlp.stanford.edu/~manning/)\n20. [ Clement Farabet ](http://www.clement.farabet.net/)\n21. [ Dan Claudiu Cireșan ](http://www.idsia.ch/~ciresan/)\n22. [ David Reichert ](http://serre-lab.clps.brown.edu/person/david-reichert/)\n23. [ Derek Rose ](http://mil.engr.utk.edu/nmil/member/5.html)\n24. [ Dong Yu ](http://research.microsoft.com/en-us/people/dongyu/default.aspx)\n25. [ Drausin Wulsin ](http://www.seas.upenn.edu/~wulsin/)\n26. [ Erik M. Schmidt ](http://music.ece.drexel.edu/people/eschmidt)\n27. [ Eugenio Culurciello ](https://engineering.purdue.edu/BME/People/viewPersonById?resource_id=71333)\n28. [ Frank Seide ](http://research.microsoft.com/en-us/people/fseide/)\n29. [ Galen Andrew ](http://homes.cs.washington.edu/~galen/)\n30. [ Geoffrey Hinton ](http://www.cs.toronto.edu/~hinton/)\n31. [ George Dahl ](http://www.cs.toronto.edu/~gdahl/)\n32. [ Graham Taylor ](http://www.uoguelph.ca/~gwtaylor/)\n33. [ Grégoire Montavon ](http://gregoire.montavon.name/)\n34. [ Guido Francisco Montúfar ](http://personal-homepages.mis.mpg.de/montufar/)\n35. [ Guillaume Desjardins ](http://brainlogging.wordpress.com/)\n36. [ Hannes Schulz ](http://www.ais.uni-bonn.de/~schulz/)\n37. [ Hélène Paugam-Moisy ](http://www.lri.fr/~hpaugam/)\n38. [ Honglak Lee ](http://web.eecs.umich.edu/~honglak/)\n39. [ Hugo Larochelle ](http://www.dmi.usherb.ca/~larocheh/index_en.html)\n40. [ Ilya Sutskever ](http://www.cs.toronto.edu/~ilya/)\n41. [ Itamar Arel ](http://mil.engr.utk.edu/nmil/member/2.html)\n42. [ James Martens ](http://www.cs.toronto.edu/~jmartens/)\n43. [ Jason Morton ](http://www.jasonmorton.com/)\n44. [ Jason Weston ](http://www.thespermwhale.com/jaseweston/)\n45. [ Jeff Dean ](http://research.google.com/pubs/jeff.html)\n46. [ Jiquan Mgiam ](http://cs.stanford.edu/~jngiam/)\n47. [ Joseph Turian ](http://www-etud.iro.umontreal.ca/~turian/)\n48. [ Joshua Matthew Susskind ](http://aclab.ca/users/josh/index.html)\n49. [ Jürgen Schmidhuber ](http://www.idsia.ch/~juergen/)\n50. [ Justin A. Blanco ](https://sites.google.com/site/blancousna/)\n51. [ Koray Kavukcuoglu ](http://koray.kavukcuoglu.org/)\n52. [ KyungHyun Cho ](http://users.ics.aalto.fi/kcho/)\n53. [ Li Deng ](http://research.microsoft.com/en-us/people/deng/)\n54. [ Lucas Theis ](http://www.kyb.tuebingen.mpg.de/nc/employee/details/lucas.html)\n55. [ Ludovic Arnold ](http://ludovicarnold.altervista.org/home/)\n56. [ Marc'Aurelio Ranzato ](http://www.cs.nyu.edu/~ranzato/)\n57. [ Martin Längkvist ](http://aass.oru.se/~mlt/)\n58. [ Misha Denil ](http://mdenil.com/)\n59. [ Mohammad Norouzi ](http://www.cs.toronto.edu/~norouzi/)\n60. [ Nando de Freitas ](http://www.cs.ubc.ca/~nando/)\n61. [ Navdeep Jaitly ](http://www.cs.utoronto.ca/~ndjaitly/)\n62. [ Nicolas Le Roux ](http://nicolas.le-roux.name/)\n63. [ Nitish Srivastava ](http://www.cs.toronto.edu/~nitish/)\n64. [ Noel Lopes ](https://www.cisuc.uc.pt/people/show/2028)\n65. [ Oriol Vinyals ](http://www.cs.berkeley.edu/~vinyals/)\n66. [ Pascal Vincent ](http://www.iro.umontreal.ca/~vincentp)\n67. [ Patrick Nguyen ](https://sites.google.com/site/drpngx/)\n68. [ Pedro Domingos ](http://homes.cs.washington.edu/~pedrod/)\n69. [ Peggy Series ](http://homepages.inf.ed.ac.uk/pseries/)\n70. [ Pierre Sermanet ](http://cs.nyu.edu/~sermanet)\n71. [ Piotr Mirowski ](http://www.cs.nyu.edu/~mirowski/)\n72. [ Quoc V. Le ](http://ai.stanford.edu/~quocle/)\n73. [ Reinhold Scherer ](http://bci.tugraz.at/scherer/)\n74. [ Richard Socher ](http://www.socher.org/)\n75. [ Rob Fergus ](http://cs.nyu.edu/~fergus/pmwiki/pmwiki.php)\n76. [ Robert Coop ](http://mil.engr.utk.edu/nmil/member/19.html)\n77. [ Robert Gens ](http://homes.cs.washington.edu/~rcg/)\n78. [ Roger Grosse ](http://people.csail.mit.edu/rgrosse/)\n79. [ Ronan Collobert ](http://ronan.collobert.com/)\n80. [ Ruslan Salakhutdinov ](http://www.utstat.toronto.edu/~rsalakhu/)\n81. [ Sebastian Gerwinn ](http://www.kyb.tuebingen.mpg.de/nc/employee/details/sgerwinn.html)\n82. [ Stéphane Mallat ](http://www.cmap.polytechnique.fr/~mallat/)\n83. [ Sven Behnke ](http://www.ais.uni-bonn.de/behnke/)\n84. [ Tapani Raiko ](http://users.ics.aalto.fi/praiko/)\n85. [ Tara Sainath ](https://sites.google.com/site/tsainath/)\n86. [ Tijmen Tieleman ](http://www.cs.toronto.edu/~tijmen/)\n87. [ Tom Karnowski ](http://mil.engr.utk.edu/nmil/member/36.html)\n88. [ Tomáš Mikolov ](https://research.facebook.com/tomas-mikolov)\n89. [ Ueli Meier ](http://www.idsia.ch/~meier/)\n90. [ Vincent Vanhoucke ](http://vincent.vanhoucke.com)\n91. [ Volodymyr Mnih ](http://www.cs.toronto.edu/~vmnih/)\n92. [ Yann LeCun ](http://yann.lecun.com/)\n93. [ Yichuan Tang ](http://www.cs.toronto.edu/~tang/)\n94. [ Yoshua Bengio ](http://www.iro.umontreal.ca/~bengioy/yoshua_en/index.html)\n95. [ Yotaro Kubo ](http://yota.ro/)\n96. [ Youzhi (Will) Zou ](http://ai.stanford.edu/~wzou)\n97. [ Fei-Fei Li ](http://vision.stanford.edu/feifeili)\n98. [ Ian Goodfellow ](https://research.google.com/pubs/105214.html)\n99. [ Robert Laganière ](http://www.site.uottawa.ca/~laganier/)\n\n\n### WebSites\n\n1. [deeplearning.net](http://deeplearning.net/)\n2. [deeplearning.stanford.edu](http://deeplearning.stanford.edu/)\n3. [nlp.stanford.edu](http://nlp.stanford.edu/)\n4. [ai-junkie.com](http://www.ai-junkie.com/ann/evolved/nnt1.html)\n5. [cs.brown.edu/research/ai](http://cs.brown.edu/research/ai/)\n6. [eecs.umich.edu/ai](http://www.eecs.umich.edu/ai/)\n7. [cs.utexas.edu/users/ai-lab](http://www.cs.utexas.edu/users/ai-lab/)\n8. [cs.washington.edu/research/ai](http://www.cs.washington.edu/research/ai/)\n9. [aiai.ed.ac.uk](http://www.aiai.ed.ac.uk/)\n10. [www-aig.jpl.nasa.gov](http://www-aig.jpl.nasa.gov/)\n11. [csail.mit.edu](http://www.csail.mit.edu/)\n12. [cgi.cse.unsw.edu.au/~aishare](http://cgi.cse.unsw.edu.au/~aishare/)\n13. [cs.rochester.edu/research/ai](http://www.cs.rochester.edu/research/ai/)\n14. [ai.sri.com](http://www.ai.sri.com/)\n15. [isi.edu/AI/isd.htm](http://www.isi.edu/AI/isd.htm)\n16. [nrl.navy.mil/itd/aic](http://www.nrl.navy.mil/itd/aic/)\n17. [hips.seas.harvard.edu](http://hips.seas.harvard.edu/)\n18. [AI Weekly](http://aiweekly.co)\n19. [stat.ucla.edu](http://www.stat.ucla.edu/~junhua.mao/m-RNN.html)\n20. [deeplearning.cs.toronto.edu](http://deeplearning.cs.toronto.edu/i2t)\n21. [jeffdonahue.com/lrcn/](http://jeffdonahue.com/lrcn/)\n22. [visualqa.org](http://www.visualqa.org/)\n23. [www.mpi-inf.mpg.de/departments/computer-vision...](https://www.mpi-inf.mpg.de/departments/computer-vision-and-multimodal-computing/)\n24. [Deep Learning News](http://news.startup.ml/)\n25. [Machine Learning is Fun! Adam Geitgey's Blog](https://medium.com/@ageitgey/)\n26. [Guide to Machine Learning](http://yerevann.com/a-guide-to-deep-learning/)\n27. [Deep Learning for Beginners](https://spandan-madan.github.io/DeepLearningProject/)\n\n### Datasets\n\n1. [MNIST](http://yann.lecun.com/exdb/mnist/) Handwritten digits\n2. [Google House Numbers](http://ufldl.stanford.edu/housenumbers/) from street view\n3. [CIFAR-10 and CIFAR-100](http://www.cs.toronto.edu/~kriz/cifar.html)\n4. [IMAGENET](http://www.image-net.org/)\n5. [Tiny Images](http://groups.csail.mit.edu/vision/TinyImages/) 80 Million tiny images6. \n6. [Flickr Data](https://yahooresearch.tumblr.com/post/89783581601/one-hundred-million-creative-commons-flickr-images) 100 Million Yahoo dataset\n7. [Berkeley Segmentation Dataset 500](http://www.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/)\n8. [UC Irvine Machine Learning Repository](http://archive.ics.uci.edu/ml/)\n9. [Flickr 8k](http://nlp.cs.illinois.edu/HockenmaierGroup/Framing_Image_Description/KCCA.html)\n10. [Flickr 30k](http://shannon.cs.illinois.edu/DenotationGraph/)\n11. [Microsoft COCO](http://mscoco.org/home/)\n12. [VQA](http://www.visualqa.org/)\n13. [Image QA](http://www.cs.toronto.edu/~mren/imageqa/data/cocoqa/)\n14. [AT&T Laboratories Cambridge face database](http://www.uk.research.att.com/facedatabase.html)\n15. [AVHRR Pathfinder](http://xtreme.gsfc.nasa.gov)\n16. [Air Freight](http://www.anc.ed.ac.uk/~amos/afreightdata.html) - The Air Freight data set is a ray-traced image sequence along with ground truth segmentation based on textural characteristics. (455 images + GT, each 160x120 pixels). (Formats: PNG) \n17. [Amsterdam Library of Object Images](http://www.science.uva.nl/~aloi/) - ALOI is a color image collection of one-thousand small objects, recorded for scientific purposes. In order to capture the sensory variation in object recordings, we systematically varied viewing angle, illumination angle, and illumination color for each object, and additionally captured wide-baseline stereo images. We recorded over a hundred images of each object, yielding a total of 110,250 images for the collection. (Formats: png)\n18. [Annotated face, hand, cardiac & meat images](http://www.imm.dtu.dk/~aam/) - Most images & annotations are supplemented by various ASM/AAM analyses using the AAM-API. (Formats: bmp,asf)\n19. [Image Analysis and Computer Graphics](http://www.imm.dtu.dk/image/) \n21. [Brown University Stimuli](http://www.cog.brown.edu/~tarr/stimuli.html) - A variety of datasets including geons, objects, and \"greebles\". Good for testing recognition algorithms. (Formats: pict)\n22. [CAVIAR video sequences of mall and public space behavior](http://homepages.inf.ed.ac.uk/rbf/CAVIARDATA1/) - 90K video frames in 90 sequences of various human activities, with XML ground truth of detection and behavior classification (Formats: MPEG2 & JPEG)\n23. [Machine Vision Unit](http://www.ipab.inf.ed.ac.uk/mvu/)\n25. [CCITT Fax standard images](http://www.cs.waikato.ac.nz/~singlis/ccitt.html) - 8 images (Formats: gif)\n26. [CMU CIL's Stereo Data with Ground Truth](cil-ster.html) - 3 sets of 11 images, including color tiff images with spectroradiometry (Formats: gif, tiff)\n27. [CMU PIE Database](http://www.ri.cmu.edu/projects/project_418.html) - A database of 41,368 face images of 68 people captured under 13 poses, 43 illuminations conditions, and with 4 different expressions.\n28. [CMU VASC Image Database](http://www.ius.cs.cmu.edu/idb/) - Images, sequences, stereo pairs (thousands of images) (Formats: Sun Rasterimage)\n29. [Caltech Image Database](http://www.vision.caltech.edu/html-files/archive.html) - about 20 images - mostly top-down views of small objects and toys. (Formats: GIF)\n30. [Columbia-Utrecht Reflectance and Texture Database](http://www.cs.columbia.edu/CAVE/curet/) - Texture and reflectance measurements for over 60 samples of 3D texture, observed with over 200 different combinations of viewing and illumination directions. (Formats: bmp)\n31. [Computational Colour Constancy Data](http://www.cs.sfu.ca/~colour/data/index.html) - A dataset oriented towards computational color constancy, but useful for computer vision in general. It includes synthetic data, camera sensor data, and over 700 images. (Formats: tiff)\n32. [Computational Vision Lab](http://www.cs.sfu.ca/~colour/)\n34. [Content-based image retrieval database](http://www.cs.washington.edu/research/imagedatabase/groundtruth/) - 11 sets of color images for testing algorithms for content-based retrieval. Most sets have a description file with names of objects in each image. (Formats: jpg) \n35. [Efficient Content-based Retrieval Group](http://www.cs.washington.edu/research/imagedatabase/)\n37. [Densely Sampled View Spheres](http://ls7-www.cs.uni-dortmund.de/~peters/pages/research/modeladaptsys/modeladaptsys_vba_rov.html) - Densely sampled view spheres - upper half of the view sphere of two toy objects with 2500 images each. (Formats: tiff)\n38. [Computer Science VII (Graphical Systems)](http://ls7-www.cs.uni-dortmund.de/)\n40. [Digital Embryos](https://web-beta.archive.org/web/20011216051535/vision.psych.umn.edu/www/kersten-lab/demos/digitalembryo.html) - Digital embryos are novel objects which may be used to develop and test object recognition systems. They have an organic appearance. (Formats: various formats are available on request)\n41. [Univerity of Minnesota Vision Lab](http://vision.psych.umn.edu/www/kersten-lab/kersten-lab.html) \n42. [El Salvador Atlas of Gastrointestinal VideoEndoscopy](http://www.gastrointestinalatlas.com) - Images and Videos of his-res of studies taken from Gastrointestinal Video endoscopy. (Formats: jpg, mpg, gif)\n43. [FG-NET Facial Aging Database](http://sting.cycollege.ac.cy/~alanitis/fgnetaging/index.htm) - Database contains 1002 face images showing subjects at different ages. (Formats: jpg)\n44. [FVC2000 Fingerprint Databases](http://bias.csr.unibo.it/fvc2000/) - FVC2000 is the First International Competition for Fingerprint Verification Algorithms. Four fingerprint databases constitute the FVC2000 benchmark (3520 fingerprints in all).\n45. [Biometric Systems Lab](http://bias.csr.unibo.it/research/biolab) - University of Bologna\n46. [Face and Gesture images and image sequences](http://www.fg-net.org) - Several image datasets of faces and gestures that are ground truth annotated for benchmarking\n47. [German Fingerspelling Database](http://www-i6.informatik.rwth-aachen.de/~dreuw/database.html) - The database contains 35 gestures and consists of 1400 image sequences that contain gestures of 20 different persons recorded under non-uniform daylight lighting conditions. (Formats: mpg,jpg) \n48. [Language Processing and Pattern Recognition](http://www-i6.informatik.rwth-aachen.de/)\n50. [Groningen Natural Image Database](http://hlab.phys.rug.nl/archive.html) - 4000+ 1536x1024 (16 bit) calibrated outdoor images (Formats: homebrew)\n51. [ICG Testhouse sequence](http://www.icg.tu-graz.ac.at/~schindler/Data) - 2 turntable sequences from ifferent viewing heights, 36 images each, resolution 1000x750, color (Formats: PPM)\n52. [Institute of Computer Graphics and Vision](http://www.icg.tu-graz.ac.at)\n54. [IEN Image Library](http://www.ien.it/is/vislib/) - 1000+ images, mostly outdoor sequences (Formats: raw, ppm) \n55. [INRIA's Syntim images database](http://www-rocq.inria.fr/~tarel/syntim/images.html) - 15 color image of simple objects (Formats: gif)\n56. [INRIA](http://www.inria.fr/)\n57. [INRIA's Syntim stereo databases](http://www-rocq.inria.fr/~tarel/syntim/paires.html) - 34 calibrated color stereo pairs (Formats: gif) \n58. [Image Analysis Laboratory](http://www.ece.ncsu.edu/imaging/Archives/ImageDataBase/index.html) - Images obtained from a variety of imaging modalities -- raw CFA images, range images and a host of \"medical images\". (Formats: homebrew)\n59. [Image Analysis Laboratory](http://www.ece.ncsu.edu/imaging) \n61. [Image Database](http://www.prip.tuwien.ac.at/prip/image.html) - An image database including some textures \n62. [JAFFE Facial Expression Image Database](http://www.mis.atr.co.jp/~mlyons/jaffe.html) - The JAFFE database consists of 213 images of Japanese female subjects posing 6 basic facial expressions as well as a neutral pose. Ratings on emotion adjectives are also available, free of charge, for research purposes. (Formats: TIFF Grayscale images.) \n63. [ATR Research, Kyoto, Japan](http://www.mic.atr.co.jp/)\n64. [JISCT Stereo Evaluation](ftp://ftp.vislist.com/IMAGERY/JISCT/) - 44 image pairs. These data have been used in an evaluation of stereo analysis, as described in the April 1993 ARPA Image Understanding Workshop paper ``The JISCT Stereo Evaluation'' by R.C.Bolles, H.H.Baker, and M.J.Hannah, 263--274 (Formats: SSI) \n65. [MIT Vision Texture](http://www-white.media.mit.edu/vismod/imagery/VisionTexture/vistex.html) - Image archive (100+ images) (Formats: ppm)\n66. [MIT face images and more](ftp://whitechapel.media.mit.edu/pub/images) - hundreds of images (Formats: homebrew) \n67. [Machine Vision](http://vision.cse.psu.edu/book/testbed/images/) - Images from the textbook by Jain, Kasturi, Schunck (20+ images) (Formats: GIF TIFF)\n68. [Mammography Image Databases](http://marathon.csee.usf.edu/Mammography/Database.html) - 100 or more images of mammograms with ground truth. Additional images available by request, and links to several other mammography databases are provided. (Formats: homebrew)\n69. [ftp://ftp.cps.msu.edu/pub/prip](ftp://ftp.cps.msu.edu/pub/prip) - many images (Formats: unknown)\n70. [Middlebury Stereo Data Sets with Ground Truth](http://www.middlebury.edu/stereo/data.html) - Six multi-frame stereo data sets of scenes containing planar regions. Each data set contains 9 color images and subpixel-accuracy ground-truth data. (Formats: ppm)\n71. [Middlebury Stereo Vision Research Page](http://www.middlebury.edu/stereo) - Middlebury College\n72. [Modis Airborne simulator, Gallery and data set](http://ltpwww.gsfc.nasa.gov/MODIS/MAS/) - High Altitude Imagery from around the world for environmental modeling in support of NASA EOS program (Formats: JPG and HDF)\n73. [NIST Fingerprint and handwriting](ftp://sequoyah.ncsl.nist.gov/pub/databases/data) - datasets - thousands of images (Formats: unknown)\n74. [NIST Fingerprint data](ftp://ftp.cs.columbia.edu/jpeg/other/uuencoded) - compressed multipart uuencoded tar file \n75. [NLM HyperDoc Visible Human Project](http://www.nlm.nih.gov/research/visible/visible_human.html) - Color, CAT and MRI image samples - over 30 images (Formats: jpeg)\n76. [National Design Repository](http://www.designrepository.org) - Over 55,000 3D CAD and solid models of (mostly) mechanical/machined engineerign designs. (Formats: gif,vrml,wrl,stp,sat) \n77. [Geometric & Intelligent Computing Laboratory](http://gicl.mcs.drexel.edu) \n79. [OSU (MSU) 3D Object Model Database](http://eewww.eng.ohio-state.edu/~flynn/3DDB/Models/) - several sets of 3D object models collected over several years to use in object recognition research (Formats: homebrew, vrml)\n80. [OSU (MSU/WSU) Range Image Database](http://eewww.eng.ohio-state.edu/~flynn/3DDB/RID/) - Hundreds of real and synthetic images (Formats: gif, homebrew)\n81. [OSU/SAMPL Database: Range Images, 3D Models, Stills, Motion Sequences](http://sampl.eng.ohio-state.edu/~sampl/database.htm) - Over 1000 range images, 3D object models, still images and motion sequences (Formats: gif, ppm, vrml, homebrew) \n82. [Signal Analysis and Machine Perception Laboratory](http://sampl.eng.ohio-state.edu)\n84. [Otago Optical Flow Evaluation Sequences](http://www.cs.otago.ac.nz/research/vision/Research/OpticalFlow/opticalflow.html) - Synthetic and real sequences with machine-readable ground truth optical flow fields, plus tools to generate ground truth for new sequences. (Formats: ppm,tif,homebrew)\n85. [Vision Research Group](http://www.cs.otago.ac.nz/research/vision/index.html) \n87. [ftp://ftp.limsi.fr/pub/quenot/opflow/testdata/piv/](ftp://ftp.limsi.fr/pub/quenot/opflow/testdata/piv/) - Real and synthetic image sequences used for testing a Particle Image Velocimetry application. These images may be used for the test of optical flow and image matching algorithms. (Formats: pgm (raw)) \n88. [LIMSI-CNRS/CHM/IMM/vision](http://www.limsi.fr/Recherche/IMM/PageIMM.html) \n89. [LIMSI-CNRS](http://www.limsi.fr/)\n90. [Photometric 3D Surface Texture Database](http://www.taurusstudio.net/research/pmtexdb/index.htm) - This is the first 3D texture database which provides both full real surface rotations and registered photometric stereo data (30 textures, 1680 images). (Formats: TIFF) \n91. [SEQUENCES FOR OPTICAL FLOW ANALYSIS (SOFA)](http://www.cee.hw.ac.uk/~mtc/sofa) - 9 synthetic sequences designed for testing motion analysis applications, including full ground truth of motion and camera parameters. (Formats: gif)\n92. [Computer Vision Group](http://www.cee.hw.ac.uk/~mtc/research.html)\n94. [Sequences for Flow Based Reconstruction](http://www.nada.kth.se/~zucch/CAMERA/PUB/seq.html) - synthetic sequence for testing structure from motion algorithms (Formats: pgm)\n95. [Stereo Images with Ground Truth Disparity and Occlusion](http://www-dbv.cs.uni-bonn.de/stereo_data/) - a small set of synthetic images of a hallway with varying amounts of noise added. Use these images to benchmark your stereo algorithm. (Formats: raw, viff (khoros), or tiff)\n96. [Stuttgart Range Image Database](http://range.informatik.uni-stuttgart.de) - A collection of synthetic range images taken from high-resolution polygonal models available on the web (Formats: homebrew)\n97. [Department Image Understanding](http://www.informatik.uni-stuttgart.de/ipvr/bv/bv_home_engl.html) \n99. [The AR Face Database](http://www2.ece.ohio-state.edu/~aleix/ARdatabase.html) - Contains over 4,000 color images corresponding to 126 people's faces (70 men and 56 women). Frontal views with variations in facial expressions, illumination, and occlusions. (Formats: RAW (RGB 24-bit))\n100. [Purdue Robot Vision Lab](http://rvl.www.ecn.purdue.edu/RVL/)\n101. [The MIT-CSAIL Database of Objects and Scenes](http://web.mit.edu/torralba/www/database.html) - Database for testing multiclass object detection and scene recognition algorithms. Over 72,000 images with 2873 annotated frames. More than 50 annotated object classes. (Formats: jpg)\n102. [The RVL SPEC-DB (SPECularity DataBase)](http://rvl1.ecn.purdue.edu/RVL/specularity_database/) - A collection of over 300 real images of 100 objects taken under three different illuminaiton conditions (Diffuse/Ambient/Directed). -- Use these images to test algorithms for detecting and compensating specular highlights in color images. (Formats: TIFF )\n103. [Robot Vision Laboratory](http://rvl1.ecn.purdue.edu/RVL/)\n105. [The Xm2vts database](http://xm2vtsdb.ee.surrey.ac.uk) - The XM2VTSDB contains four digital recordings of 295 people taken over a period of four months. This database contains both image and video data of faces.\n106. [Centre for Vision, Speech and Signal Processing](http://www.ee.surrey.ac.uk/Research/CVSSP) \n107. [Traffic Image Sequences and 'Marbled Block' Sequence](http://i21www.ira.uka.de/image_sequences) - thousands of frames of digitized traffic image sequences as well as the 'Marbled Block' sequence (grayscale images) (Formats: GIF)\n108. [IAKS/KOGS](http://i21www.ira.uka.de) \n110. [U Bern Face images](ftp://ftp.iam.unibe.ch/pub/Images/FaceImages) - hundreds of images (Formats: Sun rasterfile)\n111. [U Michigan textures](ftp://freebie.engin.umich.edu/pub/misc/textures) (Formats: compressed raw)\n112. [U Oulu wood and knots database](http://www.ee.oulu.fi/~olli/Projects/Lumber.Grading.html) - Includes classifications - 1000+ color images (Formats: ppm) \n113. [UCID - an Uncompressed Colour Image Database](http://vision.doc.ntu.ac.uk/datasets/UCID/ucid.html) - a benchmark database for image retrieval with predefined ground truth. (Formats: tiff) \n115. [UMass Vision Image Archive](http://vis-www.cs.umass.edu/~vislib/) - Large image database with aerial, space, stereo, medical images and more. (Formats: homebrew)\n116. [UNC's 3D image database](ftp://sunsite.unc.edu/pub/academic/computer-science/virtual-reality/3d) - many images (Formats: GIF)\n117. [USF Range Image Data with Segmentation Ground Truth](http://marathon.csee.usf.edu/range/seg-comp/SegComp.html) - 80 image sets (Formats: Sun rasterimage)\n118. [University of Oulu Physics-based Face Database](http://www.ee.oulu.fi/research/imag/color/pbfd.html) - contains color images of faces under different illuminants and camera calibration conditions as well as skin spectral reflectance measurements of each person.\n119. [Machine Vision and Media Processing Unit](http://www.ee.oulu.fi/mvmp/)\n121. [University of Oulu Texture Database](http://www.outex.oulu.fi) - Database of 320 surface textures, each captured under three illuminants, six spatial resolutions and nine rotation angles. A set of test suites is also provided so that texture segmentation, classification, and retrieval algorithms can be tested in a standard manner. (Formats: bmp, ras, xv)\n122. [Machine Vision Group](http://www.ee.oulu.fi/mvg)\n124. [Usenix face database](ftp://ftp.uu.net/published/usenix/faces) - Thousands of face images from many different sites (circa 994)\n125. [View Sphere Database](http://www-prima.inrialpes.fr/Prima/hall/view_sphere.html) - Images of 8 objects seen from many different view points. The view sphere is sampled using a geodesic with 172 images/sphere. Two sets for training and testing are available. (Formats: ppm)\n126. [PRIMA, GRAVIR](http://www-prima.inrialpes.fr/Prima/)\n127. [Vision-list Imagery Archive](ftp://ftp.vislist.com/IMAGERY/) - Many images, many formats\n128. [Wiry Object Recognition Database](http://www.cs.cmu.edu/~owenc/word.htm) - Thousands of images of a cart, ladder, stool, bicycle, chairs, and cluttered scenes with ground truth labelings of edges and regions. (Formats: jpg)\n129. [3D Vision Group](http://www.cs.cmu.edu/0.000000E+003dvision/)\n131. [Yale Face Database](http://cvc.yale.edu/projects/yalefaces/yalefaces.html) - 165 images (15 individuals) with different lighting, expression, and occlusion configurations.\n132. [Yale Face Database B](http://cvc.yale.edu/projects/yalefacesB/yalefacesB.html) - 5760 single light source images of 10 subjects each seen under 576 viewing conditions (9 poses x 64 illumination conditions). (Formats: PGM) \n133. [Center for Computational Vision and Control](http://cvc.yale.edu/)\n134. [DeepMind QA Corpus](https://github.com/deepmind/rc-data) - Textual QA corpus from CNN and DailyMail. More than 300K documents in total. [Paper](http://arxiv.org/abs/1506.03340) for reference.\n135. [YouTube-8M Dataset](https://research.google.com/youtube8m/) - YouTube-8M is a large-scale labeled video dataset that consists of 8 million YouTube video IDs and associated labels from a diverse vocabulary of 4800 visual entities.\n136. [Open Images dataset](https://github.com/openimages/dataset) - Open Images is a dataset of ~9 million URLs to images that have been annotated with labels spanning over 6000 categories.\n137. [Visual Object Classes Challenge 2012 (VOC2012)](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html#devkit) - VOC2012 dataset containing 12k images with 20 annotated classes for object detection and segmentation.\n138. [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) - MNIST like fashion product dataset consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. \n139. [Large-scale Fashion (DeepFashion) Database](http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion.html) - Contains over 800,000 diverse fashion images. Each image in this dataset is labeled with 50 categories, 1,000 descriptive attributes, bounding box and clothing landmarks\n140. [FakeNewsCorpus](https://github.com/several27/FakeNewsCorpus) - Contains about 10 million news articles classified using [opensources.co](http://opensources.co) types\n\n### Conferences\n\n1. [CVPR - IEEE Conference on Computer Vision and Pattern Recognition](http://cvpr2018.thecvf.com)\n2. [AAMAS - International Joint Conference on Autonomous Agents and Multiagent Systems](http://celweb.vuse.vanderbilt.edu/aamas18/)\n3. [IJCAI - \tInternational Joint Conference on Artificial Intelligence](https://www.ijcai-18.org/)\n4. [ICML - \tInternational Conference on Machine Learning](https://icml.cc)\n5. [ECML - European Conference on Machine Learning](http://www.ecmlpkdd2018.org)\n6. [KDD - Knowledge Discovery and Data Mining](http://www.kdd.org/kdd2018/)\n7. [NIPS - Neural Information Processing Systems](https://nips.cc/Conferences/2018)\n8. [O'Reilly AI Conference - \tO'Reilly Artificial Intelligence Conference](https://conferences.oreilly.com/artificial-intelligence/ai-ny)\n9. [ICDM - International Conference on Data Mining](https://www.waset.org/conference/2018/07/istanbul/ICDM)\n10. [ICCV - International Conference on Computer Vision](http://iccv2017.thecvf.com)\n11. [AAAI - Association for the Advancement of Artificial Intelligence](https://www.aaai.org)\n\n### Frameworks\n\n1. [Caffe](http://caffe.berkeleyvision.org/) \n2. [Torch7](http://torch.ch/)\n3. [Theano](http://deeplearning.net/software/theano/)\n4. [cuda-convnet](https://code.google.com/p/cuda-convnet2/)\n5. [convetjs](https://github.com/karpathy/convnetjs)\n5. [Ccv](http://libccv.org/doc/doc-convnet/)\n6. [NuPIC](http://numenta.org/nupic.html)\n7. [DeepLearning4J](http://deeplearning4j.org/)\n8. [Brain](https://github.com/harthur/brain)\n9. [DeepLearnToolbox](https://github.com/rasmusbergpalm/DeepLearnToolbox)\n10. [Deepnet](https://github.com/nitishsrivastava/deepnet)\n11. [Deeppy](https://github.com/andersbll/deeppy)\n12. [JavaNN](https://github.com/ivan-vasilev/neuralnetworks)\n13. [hebel](https://github.com/hannes-brt/hebel)\n14. [Mocha.jl](https://github.com/pluskid/Mocha.jl)\n15. [OpenDL](https://github.com/guoding83128/OpenDL)\n16. [cuDNN](https://developer.nvidia.com/cuDNN)\n17. [MGL](http://melisgl.github.io/mgl-pax-world/mgl-manual.html)\n18. [Knet.jl](https://github.com/denizyuret/Knet.jl)\n19. [Nvidia DIGITS - a web app based on Caffe](https://github.com/NVIDIA/DIGITS)\n20. [Neon - Python based Deep Learning Framework](https://github.com/NervanaSystems/neon)\n21. [Keras - Theano based Deep Learning Library](http://keras.io)\n22. [Chainer - A flexible framework of neural networks for deep learning](http://chainer.org/)\n23. [RNNLM Toolkit](http://rnnlm.org/)\n24. [RNNLIB - A recurrent neural network library](http://sourceforge.net/p/rnnl/wiki/Home/)\n25. [char-rnn](https://github.com/karpathy/char-rnn)\n26. [MatConvNet: CNNs for MATLAB](https://github.com/vlfeat/matconvnet)\n27. [Minerva - a fast and flexible tool for deep learning on multi-GPU](https://github.com/dmlc/minerva)\n28. [Brainstorm - Fast, flexible and fun neural networks.](https://github.com/IDSIA/brainstorm)\n29. [Tensorflow - Open source software library for numerical computation using data flow graphs](https://github.com/tensorflow/tensorflow)\n30. [DMTK - Microsoft Distributed Machine Learning Tookit](https://github.com/Microsoft/DMTK)\n31. [Scikit Flow - Simplified interface for TensorFlow (mimicking Scikit Learn)](https://github.com/google/skflow)\n32. [MXnet - Lightweight, Portable, Flexible Distributed/Mobile Deep Learning framework](https://github.com/dmlc/mxnet/)\n33. [Veles - Samsung Distributed machine learning platform](https://github.com/Samsung/veles)\n34. [Marvin - A Minimalist GPU-only N-Dimensional ConvNets Framework](https://github.com/PrincetonVision/marvin)\n35. [Apache SINGA - A General Distributed Deep Learning Platform](http://singa.incubator.apache.org/)\n36. [DSSTNE - Amazon's library for building Deep Learning models](https://github.com/amznlabs/amazon-dsstne)\n37. [SyntaxNet - Google's syntactic parser - A TensorFlow dependency library](https://github.com/tensorflow/models/tree/master/syntaxnet)\n38. [mlpack - A scalable Machine Learning library](http://mlpack.org/)\n39. [Torchnet - Torch based Deep Learning Library](https://github.com/torchnet/torchnet)\n40. [Paddle - PArallel Distributed Deep LEarning by Baidu](https://github.com/baidu/paddle)\n41. [NeuPy - Theano based Python library for ANN and Deep Learning](http://neupy.com)\n42. [Lasagne - a lightweight library to build and train neural networks in Theano](https://github.com/Lasagne/Lasagne)\n43. [nolearn - wrappers and abstractions around existing neural network libraries, most notably Lasagne](https://github.com/dnouri/nolearn)\n44. [Sonnet - a library for constructing neural networks by Google's DeepMind](https://github.com/deepmind/sonnet)\n45. [PyTorch - Tensors and Dynamic neural networks in Python with strong GPU acceleration](https://github.com/pytorch/pytorch)\n46. [CNTK - Microsoft Cognitive Toolkit](https://github.com/Microsoft/CNTK)\n47. [Serpent.AI - Game agent framework: Use any video game as a deep learning sandbox](https://github.com/SerpentAI/SerpentAI)\n48. [Caffe2 - A New Lightweight, Modular, and Scalable Deep Learning Framework](https://github.com/caffe2/caffe2)\n49. [deeplearn.js - Hardware-accelerated deep learning and linear algebra (NumPy) library for the web](https://github.com/PAIR-code/deeplearnjs)\n50. [TensorForce - A TensorFlow library for applied reinforcement learning](https://github.com/reinforceio/tensorforce)\n51. [Coach - Reinforcement Learning Coach by Intel® AI Lab](https://github.com/NervanaSystems/coach)\n52. [albumentations - A fast and framework agnostic image augmentation library](https://github.com/albu/albumentations)\n\n### Tools\n\n1. [Netron](https://github.com/lutzroeder/netron) - Visualizer for deep learning and machine learning models\n2. [Jupyter Notebook](http://jupyter.org) - Web-based notebook environment for interactive computing\n3. [TensorBoard](https://github.com/tensorflow/tensorboard) - TensorFlow's Visualization Toolkit\n4. [Visual Studio Tools for AI](https://visualstudio.microsoft.com/downloads/ai-tools-vs) - Develop, debug and deploy deep learning and AI solutions\n\n### Miscellaneous\n\n1. [Google Plus - Deep Learning Community](https://plus.google.com/communities/112866381580457264725)\n2. [Caffe Webinar](http://on-demand-gtc.gputechconf.com/gtcnew/on-demand-gtc.php?searchByKeyword=shelhamer&searchItems=&sessionTopic=&sessionEvent=4&sessionYear=2014&sessionFormat=&submit=&select=+)\n3. [100 Best Github Resources in Github for DL](http://meta-guide.com/software-meta-guide/100-best-github-deep-learning/)\n4. [Word2Vec](https://code.google.com/p/word2vec/)\n5. [Caffe DockerFile](https://github.com/tleyden/docker/tree/master/caffe)\n6. [TorontoDeepLEarning convnet](https://github.com/TorontoDeepLearning/convnet)\n8. [gfx.js](https://github.com/clementfarabet/gfx.js)\n9. [Torch7 Cheat sheet](https://github.com/torch/torch7/wiki/Cheatsheet)\n10. [Misc from MIT's 'Advanced Natural Language Processing' course](http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-864-advanced-natural-language-processing-fall-2005/)\n11. [Misc from MIT's 'Machine Learning' course](http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-867-machine-learning-fall-2006/lecture-notes/)\n12. [Misc from MIT's 'Networks for Learning: Regression and Classification' course](http://ocw.mit.edu/courses/brain-and-cognitive-sciences/9-520-a-networks-for-learning-regression-and-classification-spring-2001/)\n13. [Misc from MIT's 'Neural Coding and Perception of Sound' course](http://ocw.mit.edu/courses/health-sciences-and-technology/hst-723j-neural-coding-and-perception-of-sound-spring-2005/index.htm)\n14. [Implementing a Distributed Deep Learning Network over Spark](http://www.datasciencecentral.com/profiles/blogs/implementing-a-distributed-deep-learning-network-over-spark)\n15. [A chess AI that learns to play chess using deep learning.](https://github.com/erikbern/deep-pink)\n16. [Reproducing the results of \"Playing Atari with Deep Reinforcement Learning\" by DeepMind](https://github.com/kristjankorjus/Replicating-DeepMind)\n17. [Wiki2Vec. Getting Word2vec vectors for entities and word from Wikipedia Dumps](https://github.com/idio/wiki2vec)\n18. [The original code from the DeepMind article + tweaks](https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner)\n19. [Google deepdream - Neural Network art](https://github.com/google/deepdream)\n20. [An efficient, batched LSTM.](https://gist.github.com/karpathy/587454dc0146a6ae21fc)\n21. [A recurrent neural network designed to generate classical music.](https://github.com/hexahedria/biaxial-rnn-music-composition)\n22. [Memory Networks Implementations - Facebook](https://github.com/facebook/MemNN)\n23. [Face recognition with Google's FaceNet deep neural network.](https://github.com/cmusatyalab/openface)\n24. [Basic digit recognition neural network](https://github.com/joeledenberg/DigitRecognition)\n25. [Emotion Recognition API Demo - Microsoft](https://www.projectoxford.ai/demo/emotion#detection)\n26. [Proof of concept for loading Caffe models in TensorFlow](https://github.com/ethereon/caffe-tensorflow)\n27. [YOLO: Real-Time Object Detection](http://pjreddie.com/darknet/yolo/#webcam)\n28. [AlphaGo - A replication of DeepMind's 2016 Nature publication, \"Mastering the game of Go with deep neural networks and tree search\"](https://github.com/Rochester-NRT/AlphaGo)\n29. [Machine Learning for Software Engineers](https://github.com/ZuzooVn/machine-learning-for-software-engineers)\n30. [Machine Learning is Fun!](https://medium.com/@ageitgey/machine-learning-is-fun-80ea3ec3c471#.oa4rzez3g)\n31. [Siraj Raval's Deep Learning tutorials](https://www.youtube.com/channel/UCWN3xxRkmTPmbKwht9FuE5A)\n32. [Dockerface](https://github.com/natanielruiz/dockerface) - Easy to install and use deep learning Faster R-CNN face detection for images and video in a docker container.\n33. [Awesome Deep Learning Music](https://github.com/ybayle/awesome-deep-learning-music) - Curated list of articles related to deep learning scientific research applied to music\n34. [Awesome Graph Embedding](https://github.com/benedekrozemberczki/awesome-graph-embedding) - Curated list of articles related to deep learning scientific research on graph structured data\n\n\n## Awesome Deep Learning - Part II\n\nThis is a rough list of my favorite deep learning resources. It has been useful to me for learning how to do deep learning, I use it for revisiting topics or for reference.\n\n## Table of Content - Part II\n\n- [Trends](#trends)\n- [Online classes](#online-classes)\n- [Books](#books)\n- [Posts and Articles](#posts-and-articles)\n- [Practical resources](#practical-resources)\n - [Librairies and Implementations](#librairies-and-implementations)\n - [Some Datasets](#some-datasets)\n- [Other Math Theory](#other-math-theory)\n - [Gradient Descent Algorithms and optimization](#gradient-descent-algorithms-and-optimization)\n - [Complex Numbers & Digital Signal Processing](#complex-numbers-and-digital-signal-processing)\n- [Papers](#papers)\n - [Recurrent Neural Networks](#recurrent-neural-networks)\n - [Convolutional Neural Networks](#convolutional-neural-networks)\n - [Attention Mechanisms](#attention-mechanisms)\n- [YouTube and Videos](#youtube)\n- [Misc. Hubs and Links](#misc-hubs-and-links)\n- [License](#license)\n\n\n\n## Trends\n\nHere are the all-time [Google Trends](https://www.google.ca/trends/explore?date=all&q=machine%20learning,deep%20learning,data%20science,computer%20programming), from 2004 up to now, September 2017:\n\n ![](\"https://raw.githubusercontent.com/guillaume-chevalier/Awesome-Deep-Learning-Resources/master/google_trends.png\")
\n