{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.\n\n\n%matplotlib inline\nimport cv2\nfrom scipy.stats import itemfreq\nimport matplotlib\nimport matplotlib.pyplot as plt\n\nimport seaborn as sns # visualizations\n\nimport random # for setting seed\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"840003a44c6882a988d9cd6263b4155c86b11ca0"},"cell_type":"markdown","source":"# Goal: Create a model that can identify an artist given a painting from the [\"Painter By Numbers\" dataset](https://www.kaggle.com/c/painter-by-numbers)\n\n# Methods / Tools\n1. [Keras](https://keras.io/)\n2. [Tensorflow](https://www.tensorflow.org/)\n3. [ResNet50](https://keras.io/applications/#resnet50)\n  * a Convolutional Neural Network (CNN) model instance in Keras\n4. [Transfer Learning](https://www.kaggle.com/dansbecker/transfer-learning)\n\n# Questions\n## 1. Can a ResNet50 model be modified & retrained to classify *Picasso* vs *Not-Picasso*?\n* similar to [hot dogs or not hot dogs](https://www.youtube.com/watch?v=ACmydtFDTGs)\n\n"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"_uuid":"26b0d572dee953c67a3d671cef9ccb0b1f03b8ac"},"cell_type":"code","source":"# #### miscellaneous questions\n# >2. Is it possible to identify an artist from the patterns of pixels in an image of a painting using the methods/tools listed above? \n# >  * __If yes:__ perhaps a CNN can process simple visual patterns (like vertical lines or horizontal lines) and using that understanding, begin to process more complex visual patterns (like shapes) then finally understand even more abstract visual patterns (like people or [hot dogs or not hot dogs](https://www.youtube.com/watch?v=ACmydtFDTGs)) \n# >  * __If no:__ well... hmmm... see question #2\n# >3. How does a CNN work to categorize images?\n# >4. Considering that ResNet50 was trained on images from the [ImageNet database](http://www.image-net.org/), would the \"low-level\" visual patterns that it learned from those images still transfer over into the domain of paintings? Why or why not and to what degree? Could the model benefit from learning from scratch, rather than by transfer learning? ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cc14234830c2211530121f3ea7b82d834977044d"},"cell_type":"code","source":"import tensorflow\nimport keras","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a47f3f2ae34da5c3ff3747a21b7a5c7752e9c4bb"},"cell_type":"markdown","source":"# First and foremost set a random seed and get environment info for the sake of [reproducibility](https://www.kaggle.com/rtatman/reproducible-research-best-practices-jupytercon?utm_medium=blog&utm_source=wordpress&utm_campaign=reproducibility-guide)"},{"metadata":{"trusted":true,"_uuid":"203e70c989b48533a3f4cf3ba0cd8982730700ae"},"cell_type":"code","source":"my_seed = 42 # 480 could work too\nrandom.seed(my_seed)\nnp.random.seed(my_seed)\ntensorflow.set_random_seed(my_seed)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"041e25c9e1294a3b4b60d562af7c1ca64c3b955d"},"cell_type":"code","source":"import IPython\n\n# print system information (but not packages)\nprint(IPython.sys_info())\n\n# get module information\n!pip freeze > frozen-requirements.txt\n\n# append system information to file\nwith open(\"frozen-requirements.txt\", \"a\") as file:\n    file.write(IPython.sys_info())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a16ec52a1a5d9cb7d5d956d69b4fd0886aa1812e"},"cell_type":"markdown","source":"# Info about [ResNet50](https://keras.io/applications/#resnet50) on Keras\n### arguments:\n* \"The default **input** size for this model is **224x224**.\"\n* \"**input_tensor**: optional Keras tensor (i.e. output of layers.Input()) to use as image input for the model.\"\n* \"**classes**: optional number of classes to classify images into, only to be specified if include_top is  True, and if no weights argument is specified.\""},{"metadata":{"_uuid":"0d6f5ebba9e9f1ea752fa222262e110af8404739"},"cell_type":"markdown","source":"# let's have a look at [Picasso](https://www.wikiart.org/en/pablo-picasso/)\n### He has 48 paintings inside train_2"},{"metadata":{"trusted":true,"_uuid":"cbf4af47d23674d1c40e21c1b057da416d123cd3","_kg_hide-input":true},"cell_type":"code","source":"filename = '2640.jpg'\ndata_dir = '../input/painter-by-numbers/'\ntrain_dir = data_dir + 'train_2/'\nimg = cv2.imread(train_dir + filename)\nimg = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\nplt.imshow(img)\n\ndf = pd.read_csv(data_dir + 'train_info.csv')\nprint(\"there are \" + str(df.shape[0]) + \" paintings inside train_2\") \n# get a dataframe that has rows referring to files starting with 2\n# because we only have those files downloaded currently\n# eg. '2.jpg' or '2640.jpg'\nmask = (df['filename'].str.startswith('2'))\ntrain_2_df = df[mask]\n\n# string of just the artist's hash code\nimg_artist = train_2_df[(train_2_df['filename'] == filename)].artist.get_values()[0]\n\nartist_data = train_2_df[(train_2_df['artist'] == img_artist)]\nnot_artist_data = train_2_df[(train_2_df['artist'] != img_artist)]\n\nnum_artist = len(artist_data)\nprint(\"Picasso has \" + str(num_artist) + \" paintings inside train_2\")\n\nartist_data.head(5)\n\n# just some code to find 2640.jpg\n#mask = [n[4:] == '.jpg' for n in df[(df['artist'] == '1950e9aa6ad878bc2a330880f77ae5a1')].filename]\n#[n for n in df[(df['artist'] == '1950e9aa6ad878bc2a330880f77ae5a1')].filename.where(mask) if type(n) == type(\"string\")]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b59785953d6cebed83f6dfe6efabd153c4a01c2c"},"cell_type":"markdown","source":"# Ok, now let's put *Picasso* images into a folder and *Not-Picasso* images into another folder"},{"metadata":{"trusted":true,"_uuid":"4ca51a54188b499d2a1d27c00e73e19400798b6b","_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"working_train_dir = \"train/\"\nworking_test_dir = \"test/\"\nif (os.path.isdir(working_train_dir) == False):\n    os.mkdir(working_train_dir)\n    print(\"created \" + working_train_dir)\nelse:\n    print(working_train_dir + \" exists\")\nif (os.path.isdir(working_test_dir) == False):\n    os.mkdir(working_test_dir)\n    print(\"created \" + working_test_dir)\nelse:\n    print(working_test_dir + \" exists\")\n\nartist_dir = working_train_dir + 'picasso/'\nnot_artist_dir = working_train_dir + 'not-picasso/'\nif (os.path.isdir(artist_dir) == False):\n    os.mkdir(artist_dir)\n    print(\"created \" + artist_dir)\nelse:\n    print(artist_dir + \" exists\")\nif (os.path.isdir(not_artist_dir) == False):\n    os.mkdir(not_artist_dir)\n    print(\"created \" + not_artist_dir)\nelse:\n    print(not_artist_dir + \" exists\")\n\n# same for test data \ntest_artist_dir = working_test_dir + 'picasso/'\ntest_not_artist_dir = working_test_dir + 'not-picasso/'\nif (os.path.isdir(test_artist_dir) == False):\n    os.mkdir(test_artist_dir)\n    print(\"created \" + test_artist_dir)\nelse:\n    print(test_artist_dir + \" exists\")\nif (os.path.isdir(test_not_artist_dir) == False):\n    os.mkdir(test_not_artist_dir)\n    print(\"created \" + test_not_artist_dir)\nelse:\n    print(test_not_artist_dir + \" exists\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c664196cc6f9cf238ec503bc3d8ea63586ca8d55","_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"from shutil import copy2\n\nnum_for_test = 10\n\ni=0\nnum_artist_in_working_dir = len([name for name in os.listdir(artist_dir)])\nfor f in artist_data['filename']:\n    len_dir = len([name for name in os.listdir(artist_dir)]) # to do make this more efficient\n    len_test_dir = len([name for name in os.listdir(test_artist_dir)])\n    if (len_dir >= num_artist - num_for_test):\n        if (len_test_dir >= num_for_test):\n            break\n        if (os.path.exists(train_dir+f) and not os.path.exists(test_artist_dir+f)):\n            copy2(train_dir+f, test_artist_dir)\n            i+=1\n    elif (os.path.exists(train_dir+f) and not os.path.exists(artist_dir+f)):\n        copy2(train_dir+f, artist_dir)\n        i+=1\n    else:\n        None #print(str(i), end=\" \")\n\nprint(\"\\ncopied artist_data \" + str(i))\n\ni=0\nnum_not_artist_in_working_dir = len([name for name in os.listdir(not_artist_dir)])\nfor f in not_artist_data['filename']:\n    len_dir = len([name for name in os.listdir(not_artist_dir)]) # to do make this more efficient\n    len_test_dir = len([name for name in os.listdir(test_not_artist_dir)])\n    if (len_dir >= num_artist - num_for_test):\n        if (len_test_dir >= num_for_test):\n            break\n        if (os.path.exists(train_dir+f) and not os.path.exists(test_not_artist_dir+f)):\n            copy2(train_dir+f, test_not_artist_dir)\n            i+=1\n    elif (os.path.exists(train_dir+f) and not os.path.exists(not_artist_dir+f)):\n        copy2(train_dir+f, not_artist_dir)\n        i+=1\n    else:\n        None #print(str(i), end=\" \")\n\nprint(\"\\ncopied not_artist_data \" + str(i))\n# # # check if files are in the dirs\n# # for f in artist_data['filename']:\n# #     print(f+'\\t', str(os.path.exists(train_dir+f))+'\\t\\t', os.path.exists('picasso/'+f))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"b90bb4ea4e456a3fc0d8594f7d5a9519cc3065de"},"cell_type":"code","source":"print(artist_dir+'\\t\\t', len(os.listdir(artist_dir)))\nprint(test_artist_dir+'\\t\\t', len(os.listdir(test_artist_dir)))\nprint(not_artist_dir+'\\t', len(os.listdir(not_artist_dir)))\nprint(test_not_artist_dir+'\\t', len(os.listdir(test_not_artist_dir)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fb71dd2152115a8189afe83f0bf7c835544eec71"},"cell_type":"markdown","source":"# Let's try following [this lesson on transfer learning](https://www.kaggle.com/dansbecker/transfer-learning)"},{"metadata":{"_uuid":"e42b4a1f9340e962c9330796cb917916165385ad"},"cell_type":"markdown","source":"## Specify Model"},{"metadata":{"trusted":true,"_uuid":"1fd0c2c0ca0fccb8f973ecb210b865d8c01bfb15"},"cell_type":"code","source":"from tensorflow.python.keras.applications import ResNet50\nfrom tensorflow.python.keras.models import Sequential\nfrom tensorflow.python.keras.layers import Dense, Flatten, GlobalAveragePooling2D\n\nnum_classes = 2 # picasso or not picasso\n\n# this file is the resnet50 model trained on ImageNet data...\n# \"notop\" means the file does not include weights for the last layer (prediction layer)\n# in order to allow for transfer learning\nweights_notop_path = '../input/resnet50/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5'\n\n# declare new Sequential model\n# meaning each layer is in sequence, one after the other. \n# apparently there can be non-sequential neural networks... wow!\nmodel = Sequential()\n\n# now let's set up the first layers\nmodel.add(ResNet50(    # add a whole ResNet50 model\n  include_top=False,          # without the last layer\n  weights=weights_notop_path, # and with the \"notop\" weights file\n  pooling='avg' # means collapse extra \"channels\" into 1D tensor by taking an avg across channels\n))\n\n\n# Now lets add a \"Dense\" layer to make predictions\nmodel.add(Dense(\n  num_classes, # this last layer just has 2 nodes\n  activation='softmax' # apply softmax function to turn values of this layer into probabilities\n))\n\n# do not train the first layer\n# because it is already smart\n# it learned cool patterns from ImageNet\nmodel.layers[0].trainable = False\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2bfdaa80ce7c70310835e6fe4c789bbd7978c7a4"},"cell_type":"markdown","source":"## Compile Model\n### TODO: learn what it means to \"***compile***\""},{"metadata":{"trusted":true,"_uuid":"c75affcd0224baf0e006e0cb12a4785fc84a4d1a"},"cell_type":"code","source":"model.compile(\n  optimizer='sgd', # stochastic gradient descent (how to update Dense connections during training)\n  loss='categorical_crossentropy', # aka \"log loss\" -- the cost function to minimize \n  # so 'optimizer' algorithm will minimize 'loss' function\n  metrics=['accuracy'] # ask it to report % of correct predictions\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dc1f6d90f1ad532978b1b76db8f39019a7b1ed99"},"cell_type":"markdown","source":"## Fitting the Model"},{"metadata":{"trusted":true,"_uuid":"51b4b1c67b5c0747c3b2ed8691ddad26eb22ab6b"},"cell_type":"code","source":"from tensorflow.python.keras.applications.resnet50 import preprocess_input\nfrom tensorflow.python.keras.preprocessing.image import ImageDataGenerator\n\nimage_size = 224\n\ndata_generator_no_aug = ImageDataGenerator(preprocessing_function=preprocess_input)\n\ntrain_generator_no_aug = data_generator_no_aug.flow_from_directory(\n        working_train_dir,\n        target_size=(image_size, image_size),\n        batch_size=24,\n        class_mode='categorical')\n\nvalidation_generator = data_generator_no_aug.flow_from_directory(\n        working_test_dir,\n        target_size=(image_size, image_size),\n        class_mode='categorical')\n\nprint(\"\\n\\nmodel - train_generator_no_aug\")\n\nhistory = model.fit_generator(\n        train_generator_no_aug,\n        steps_per_epoch=3,\n        validation_data=validation_generator,\n        validation_steps=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c4fbf49a2522761558dc85dd177ed35760e24e23"},"cell_type":"markdown","source":"# This is how to [export the \"model.h5\"](https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model) file\n> **Click** \"Output\" on this Kernel and **then scroll** all the way down past the images until you see the \".h5\" files"},{"metadata":{"trusted":true,"_uuid":"52bbc0bcb82ba0b7b7f65337dfa29a8351588429"},"cell_type":"code","source":"# this line is probably all that is needed\nmodel.save('picasso_one_epoch.h5')\n\n# I chose to save the weights to see the difference in filesize\nmodel.save_weights('picasso_one_epoch_weights.h5')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa6c4e971fb52cc35b5e0642b2e73e422280b582"},"cell_type":"markdown","source":"# Try to overfit the model"},{"metadata":{"trusted":true,"_uuid":"8bad08d46f73dcc9e7151adca8780a143d9ce89f"},"cell_type":"code","source":"history_overfit = model.fit_generator(\n        train_generator_no_aug,\n        steps_per_epoch=3,\n        epochs=8, # so... total of 9 epochs?\n        validation_data=validation_generator,\n        validation_steps=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c9495da36a2d095fa152b046251489f66e6f10ba"},"cell_type":"markdown","source":"# A plot of the ~~```learning```~~ memorization process\n* It's hard to say the model \"learned\" a general concept of what makes a Picasso painting, given a small dataset of only 38 *Picasso* images and 38 *Not-Picasso* images to train on. "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"657cd30277eec74c56181b01a303fc0361580154"},"cell_type":"code","source":"# Plot training & validation accuracy values\nplt.plot(history_overfit.history['acc'])\nplt.plot(history_overfit.history['val_acc'])\nplt.title('Model accuracy')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Test'], loc='upper left')\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history_overfit.history['loss'])\nplt.plot(history_overfit.history['val_loss'])\nplt.title('Model loss')\nplt.ylabel('Loss')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Test'], loc='upper left')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"829ed498ced8b2e6dc556768c786e1f526b003ba"},"cell_type":"markdown","source":"# Looking at the plot above\n* What would cause the orange graph of the loss function to ever spike or increase? \n  * Loss function is expected to descrease over time with gradient descent\n  * perhaps because this is a very very small subset of the whole dataset\n* What would cause the model accuracy to ever decrease when overfitting?\n* What would it take to memorize the Picasso images with 100% accuracy? "},{"metadata":{"_uuid":"1088feafba90844ff866ebaa97a0d0afe4293f8d"},"cell_type":"markdown","source":"# Save the overfit model too\n> **Click** \"Output\" on this Kernel and **then scroll** all the way down past the images until you see the \".h5\" files"},{"metadata":{"trusted":true,"_uuid":"332f3c6958bf6c4181bbffb960e8cc1f9a85020a"},"cell_type":"code","source":"# this line is probably all that is needed\nmodel.save('picasso_overfit.h5')\n\n# I chose to save the weights to see the difference in filesize\nmodel.save_weights('picasso_overfit_weights.h5')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"56d3f645ba06c584d0447f961628895e232aba50"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"fa7f111e5ec7456dc1e35096d22047fbfa10dd7a"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"f76594a6f7fdc6fba320ff71402285ec6cc89640"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"41d7c1f185a7112fd09c89d51dbad8a9aee2c429"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"655751d1bfb0f620139020a3a271973c22dbace4"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"806b5ac90bdc9b8dce817b939a9a6cf38241b359"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"07094f93196367f8071ddd8e8541c6a3a60189ef"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"743ffbacde7837603bb5bcf0686f5ea07fb0672e"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"31447c1e4392f65ffbdd78930c2ff4e3c3f05454"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"362e7e348954d8914b014e8f788147d16f806f90"},"cell_type":"markdown","source":"## Fitting a Model With [Data Augmentation](https://www.kaggle.com/dansbecker/data-augmentation/)\n### seems like the validation accuracy is worse... maybe data augmentation is better for larger datasets\n### after all Picasso only has 38 training and 10 test paintings\n### and apparently there is \"luck\" involved with small datasets"},{"metadata":{"trusted":true,"_uuid":"30d792475f2a7573a666d15c1e595a22f532b988"},"cell_type":"code","source":"data_generator_with_aug = ImageDataGenerator(preprocessing_function=preprocess_input,\n                                   horizontal_flip=True,\n                                   width_shift_range = 0.2,\n                                   height_shift_range = 0.2)\n\ntrain_generator_with_aug = data_generator_with_aug.flow_from_directory(\n        working_train_dir,\n        target_size=(image_size, image_size),\n        batch_size=24,\n        class_mode='categorical')\n\ntrain_generator_small_batch = data_generator_no_aug.flow_from_directory(\n        working_train_dir,\n        target_size=(image_size, image_size),\n        batch_size=1,\n        class_mode='categorical')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"729145c6a8ef30eadd27ed487343315373a5938d"},"cell_type":"markdown","source":"## Try different models with different hyperparameters"},{"metadata":{"trusted":true,"_uuid":"ba331683cbfb908c243a90082d46ae6c637e7726"},"cell_type":"code","source":"#--------------------\nmodel2 = Sequential()\nmodel3 = Sequential()\nmodel4 = Sequential()\n#--------------------","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b1042ef8494368ca4c4ae4c972e195bc97e490a5"},"cell_type":"markdown","source":"## setup"},{"metadata":{"trusted":true,"_uuid":"89ed6ce052b9126ab02675653b381510e1407e82"},"cell_type":"code","source":"#--------------------\nmodel2.add(ResNet50(    # add a whole ResNet50 model\n  include_top=False,          # without the last layer\n  weights=weights_notop_path, # and with the \"notop\" weights file\n  pooling='avg' # means collapse extra \"channels\" into 1D tensor by taking an avg across channels\n))\nmodel3.add(ResNet50(    # add a whole ResNet50 model\n  include_top=False,          # without the last layer\n  weights=weights_notop_path, # and with the \"notop\" weights file\n  pooling='avg' # means collapse extra \"channels\" into 1D tensor by taking an avg across channels\n))\nmodel4.add(ResNet50(    # add a whole ResNet50 model\n  include_top=False,          # without the last layer\n  weights=weights_notop_path, # and with the \"notop\" weights file\n  pooling='avg' # means collapse extra \"channels\" into 1D tensor by taking an avg across channels\n))\n#--------------------\nmodel2.add(Dense(\n  num_classes, # this last layer just has 2 nodes\n  activation='softmax' # apply softmax function to turn values of this layer into probabilities\n))\nmodel3.add(Dense(\n  num_classes, # this last layer just has 2 nodes\n  activation='softmax' # apply softmax function to turn values of this layer into probabilities\n))\nmodel4.add(Dense(\n  num_classes, # this last layer just has 2 nodes\n  activation='softmax' # apply softmax function to turn values of this layer into probabilities\n))\n#--------------------\nmodel2.layers[0].trainable = False\nmodel3.layers[0].trainable = False\nmodel4.layers[0].trainable = False\n#--------------------","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ad692a695016d933b1e03a8061c0173db5c459b"},"cell_type":"markdown","source":"## compile"},{"metadata":{"trusted":true,"_uuid":"dc8845d1a0a4860a6be08205b4208dda51eeccd7"},"cell_type":"code","source":"model2.compile(\n  optimizer='sgd', # stochastic gradient descent (how to update Dense connections during training)\n  loss='categorical_crossentropy', # aka \"log loss\" -- the cost function to minimize \n  # so 'optimizer' algorithm will minimize 'loss' function\n  metrics=['accuracy'] # ask it to report % of correct predictions\n)\n\nmodel3.compile(\n  optimizer='sgd', # stochastic gradient descent (how to update Dense connections during training)\n  loss='categorical_crossentropy', # aka \"log loss\" -- the cost function to minimize \n  # so 'optimizer' algorithm will minimize 'loss' function\n  metrics=['accuracy'] # ask it to report % of correct predictions\n)\n\nmodel4.compile(\n            optimizer='adam',\n  loss='categorical_crossentropy', # aka \"log loss\" -- the cost function to minimize \n  # so 'optimizer' algorithm will minimize 'loss' function\n  metrics=['accuracy'] # ask it to report % of correct predictions\n)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ea450b7e4d04dda4cadff6e62b7f7776af1d12da"},"cell_type":"markdown","source":"## train"},{"metadata":{"trusted":true,"_uuid":"ab7ad1ab067328c2ba72187274ef3465a20e20e8"},"cell_type":"code","source":"#--------------------\nprint(\"\\n\\nmodel2 - train_generator_with_aug\")\n\nmodel2.fit_generator(\n        train_generator_with_aug,\n        steps_per_epoch=3,\n        epochs=2, # data aug allows more epochs with less worry of overfitting\n        validation_data=validation_generator,\n        validation_steps=1)\n\nprint(\"\\n\\nmodel3 - train_generator_small_batch\")\n\nmodel3.fit_generator(\n        train_generator_small_batch,\n        steps_per_epoch=3,\n        validation_data=validation_generator,\n        validation_steps=1)\n\nprint(\"\\n\\nmodel4 - optimizer=adam\")\n\nmodel4.fit_generator(\n        train_generator_no_aug,\n        steps_per_epoch=3,\n        validation_data=validation_generator,\n        validation_steps=1)\n#--------------------\n#--------------------\n#--------------------","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}