{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# DeepDream\n*\nDeepDream uses a convolutional neural network to find and enhance patterns in images via algorithmic pareidolia, thus creating a dream-like hallucinogenic appearance in the deliberately over-processed images [(wiki)](https://en.wikipedia.org/wiki/DeepDream). *\n\nIn this kernel I will explore DeepDream by using a both an InceptionV3 and a VGG-16 pretrained model in Keras. The choice of your convnet will affect your visualisation, as different architectures result in different learned features. Inception for example has been trained on (amongst others) many images of animals and the use of this convnet in a DeepDream often outputs pictures with a lot of eye-like features. By playing around with the different architectures, I will try to understand more on how they work. \n\nInspiration for this kernel were kernels by [Carlo Alberto](https://www.kaggle.com/carloalbertobarbano/convolutional-network-visualizations-deep-dream/notebook) and by [Paul Mooney](https://www.kaggle.com/paultimothymooney/pre-trained-pytorch-monkeys-a-deep-dream), which both use PyTorch .  Most of the code is from keras/deepdream found on the [github](https://github.com/keras-team/keras/blob/master/examples/deep_dream.py) of the Keras Team. I've also used the book Deep Learning with Python by François Chollet, which gives insights on how the DeepDream algorithm (and deep learning in general) works.\n\nIn the case of DeepDream an arbitrary image is fed to the network and is analyzed. The activation of an layer is maximized and the networks is asked to enhance whatever it detected. Each layer of the network deals with features at a different level of abstraction, so the complexity of features we generate depends on which layer we choose to enhance ([source](https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html )).\n\n\nIn short: \n\n- Load the original image.\n- Define a number of processing scales (i.e. image shapes), from smallest to largest.\n- Resize the original image to the smallest scale.\n- For every scale, starting with the smallest (i.e. current one):\n    - Run gradient ascent\n    - Upscale image to the next scale\n    - Reinject the detail that was lost at upscaling time\n- Stop when we are back to the original size.\n\nTo obtain the detail lost during upscaling, we simply take the original image, shrink it down, upscale it, and compare the result to the (resized) original image.\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport scipy\nimport PIL.Image\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom keras.preprocessing.image import load_img, save_img, img_to_array\nfrom keras.applications import inception_v3\nfrom keras import backend as K\n\n# To fix FailedPreconditionError:\nsess = tf.InteractiveSession()\nwith tf.Session() as sess:\n     sess.run(tf.global_variables_initializer()) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5e104558c5dce0c97283de2a7e25497849da452"},"cell_type":"markdown","source":"## InceptionV3\n\nWhen Google released DeepDream, it was based on an Inception model - now famous for its psychedelic output. Let's start with an InceptionV3. \n\nFirst, we are not going to train the model ourselves, so we want to disable all training specific operations. Then the model can be loaded. Note that this builds the Inception V3 network without its convolutional base. A dictionary is set with the names of the layers for which we try to maximize activation, as well as their weight in the final loss we try to maximize. Then we set a function to load, resize and convert the image and a function for the opposite. "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"3f8c88cc9dbbed18b2b127f87b16606ec1b7f4be"},"cell_type":"code","source":"# Disable all training specific operations\nK.set_learning_phase(0)\n\n\n# The model will be loaded with pre-trained inceptionv3 weights.\nmodel = inception_v3.InceptionV3(weights='../input/inceptionv3/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5',\n                                 include_top=False)\ndream = model.input\nprint('Model loaded.')\n\n\n# You can tweak these setting to obtain new visual effects.\nsettings = {\n    'features': {\n        'mixed2': 0.2,\n        'mixed3': 0.5,\n        'mixed4': 2.,\n        'mixed5': 1.5,\n    },\n}\n\n\n# Set a function to load, resize and convert the image. \ndef preprocess_image(image_path):\n    # Util function to open, resize and format pictures\n    # into appropriate tensors.\n    img = load_img(image_path)\n    img = img_to_array(img)\n    img = np.expand_dims(img, axis=0)\n    img = inception_v3.preprocess_input(img)\n    return img\n\n\n# And a function to do the opposite: convert a tensor into an image. \ndef deprocess_image(x):\n    # Util function to convert a tensor into a valid image.\n    if K.image_data_format() == 'channels_first':\n        x = x.reshape((3, x.shape[2], x.shape[3]))\n        x = x.transpose((1, 2, 0))\n    else:\n        x = x.reshape((x.shape[1], x.shape[2], 3))\n    x /= 2.\n    x += 0.5\n    x *= 255.\n    x = np.clip(x, 0, 255).astype('uint8')\n    return x\n\n\n# Set a dictionary that maps the layer name to the layer instance. \n# get the symbolic outputs of each \"key\" layer (we gave them unique names).\nlayer_dict = dict([(layer.name, layer) for layer in model.layers])\n\n\n# Define the loss. The way this works is first the scalar variable *loss* is set. \n# Then the loss will be defined by adding layer contributions to this variable. \nloss = K.variable(0.)\n\nfor layer_name in settings['features']:\n    # Add the L2 norm of the features of a layer to the loss.\n    assert (layer_name in layer_dict.keys(),\n            'Layer ' + layer_name + ' not found in model.')\n    coeff = settings['features'][layer_name]\n    x = layer_dict[layer_name].output\n    # We avoid border artifacts by only involving non-border pixels in the loss.\n    scaling = K.prod(K.cast(K.shape(x), 'float32'))\n    if K.image_data_format() == 'channels_first':\n        loss += coeff * K.sum(K.square(x[:, :, 2: -2, 2: -2])) / scaling\n    else:\n        loss += coeff * K.sum(K.square(x[:, 2: -2, 2: -2, :])) / scaling","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d4aec825ff237c400c4f95c518fbc9e07d4b5d66"},"cell_type":"markdown","source":"In order to do a gradient-ascent with respect to the loss, we compute the gradients, normalize and create a function for this process (so given an image, retrieve the value of the loss and gradients). We define the gradient ascent function over a number of iterations. "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"88c8b015d694a4180ddea75c00cab416dc4919e8"},"cell_type":"code","source":"# Compute the gradients of the dream wrt the loss.\ngrads = K.gradients(loss, dream)[0]\n# Normalize gradients.\ngrads /= K.maximum(K.mean(K.abs(grads)), K.epsilon())\n\n# Set up function to retrieve the value of the loss and gradients given an input image.\noutputs = [loss, grads]\nfetch_loss_and_grads = K.function([dream], outputs)\n\ndef eval_loss_and_grads(x):\n    outs = fetch_loss_and_grads([x])\n    loss_value = outs[0]\n    grad_values = outs[1]\n    return loss_value, grad_values\n\n# Helper funtion to resize\ndef resize_img(img, size):\n    img = np.copy(img)\n    if K.image_data_format() == 'channels_first':\n        factors = (1, 1,\n                   float(size[0]) / img.shape[2],\n                   float(size[1]) / img.shape[3])\n    else:\n        factors = (1,\n                   float(size[0]) / img.shape[1],\n                   float(size[1]) / img.shape[2],\n                   1)\n    return scipy.ndimage.zoom(img, factors, order=1)\n\n\n# Define the gradient ascent function over a number of iterations. \ndef gradient_ascent(x, iterations, step, max_loss=None):\n    for i in range(iterations):\n        loss_value, grad_values = eval_loss_and_grads(x)\n        if max_loss is not None and loss_value > max_loss:\n            break\n        print('..Loss value at', i, ':', loss_value)\n        x += step * grad_values\n    return x\n\n\n# Set hyperparameters. The ocatave_scale is the ratio between each successive scale (remember the upscaling mentioned before?). \n# Playing with these hyperparameters will also allow you to achieve new effects\nstep = 0.008  # Gradient ascent step size\nnum_octave = 5  # Number of scales at which to run gradient ascent\noctave_scale = 1.4  # Size ratio between scales\niterations = 20  # Number of ascent steps per scale\nmax_loss = 10.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cfb86e2e77b90066a293d258acc7547db0668c4f"},"cell_type":"markdown","source":"Choose an image and show. Let's take an surrealistic painting by Dali and see if the weirdness can be turned up a notch.."},{"metadata":{"trusted":true,"_uuid":"b0a5650b5310436363d6124504742a2c3ac14099","_kg_hide-input":true},"cell_type":"code","source":"base_image_path = \"../input/painter-by-numbers/train_2/21500.jpg\"\nimg = PIL.Image.open(base_image_path)\nimg\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"12af946870f0eca145e673dd287d6cb67d2e0762"},"cell_type":"markdown","source":"And then reshape, resize and process with the help of the functions above. "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f8cdfeab7631fe5350defcf1fa7b5f45d77a6986"},"cell_type":"code","source":"img = preprocess_image(base_image_path)\nif K.image_data_format() == 'channels_first':\n    original_shape = img.shape[2:]\nelse:\n    original_shape = img.shape[1:3]\nsuccessive_shapes = [original_shape]\nfor i in range(1, num_octave):\n    shape = tuple([int(dim / (octave_scale ** i)) for dim in original_shape])\n    successive_shapes.append(shape)\nsuccessive_shapes = successive_shapes[::-1]\noriginal_img = np.copy(img)\nshrunk_original_img = resize_img(img, successive_shapes[0])\n\nfor shape in successive_shapes:\n    print('Processing image shape', shape)\n    img = resize_img(img, shape)\n    img = gradient_ascent(img,\n                          iterations=iterations,\n                          step=step,\n                          max_loss=max_loss)\n    upscaled_shrunk_original_img = resize_img(shrunk_original_img, shape)\n    same_size_original = resize_img(original_img, shape)\n    lost_detail = same_size_original - upscaled_shrunk_original_img\n\n    img += lost_detail\n    shrunk_original_img = resize_img(original_img, shape)\n\nsave_img('dream.jpg',deprocess_image(np.copy(img)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0333465a2b7accf371a5064bf0d8eecff174c28"},"cell_type":"markdown","source":"Let's check out the first 'dream'.  There are already some eye-like features popping up, and it looks a lot more psychedelic than before, but it's not the nicest dream... "},{"metadata":{"trusted":true,"_uuid":"b8c70d08551ffaf1e4116af22e96b947d1657f8d"},"cell_type":"code","source":"dreamout = PIL.Image.open('dream.jpg')\ndreamout","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e83785fa73155272b96fdfffa47259a029b32dfb"},"cell_type":"markdown","source":"## Next up\n\nSo this was the first image. In next versions I will check different hyperparameters, use the VGG16 and take a look at combining images (Neural style transfer). Also I will check out some smaller images, as the InceptionV3 is trained on images smaller than the one used here and this might have some effect. Stay tuned. "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}