{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Dealing with Notebook Exceeded Allowed Compute when doing TTA\n## Or, dealing with Memory Limit exceeded in TensorFlow\n\nDuring recent competition Cassava Leaf Diseases Classification I've struggled quite a bit overcoming the \"Notebook Exceeded Allowed Compute\" error, so I wanted to share what helped me.\n\nThe error could be two things - either disk or RAM used up. It was definitely not disk since I wasn't writing anything except for a small CSV file.\n\nAs for the memory usage..."},{"metadata":{},"cell_type":"markdown","source":"Here's my code without TTA that simply iterates through examples, no TTA.\n(I was using a script and not a notebook, so only excerpts here)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"def run_predictions(model):\n    predictions = []\n    test_image_names = []\n\n    test_dir = os.path.join(INPUT_FOLDER, \"test_images\")\n\n    for image_name in os.listdir(test_dir):\n        image = tf.keras.preprocessing.image.load_img(os.path.join(test_dir, image_name), target_size=IMAGE_SIZE)\n        image = tf.keras.preprocessing.image.img_to_array(image)\n        image = np.expand_dims(image, axis=0)\n        image = tf.keras.applications.inception_v3.preprocess_input(image)\n\n        cur_prediction = model.predict(image)\n\n        predictions.append(cur_prediction[0])\n        test_image_names.append(image_name)\n\n    return predictions, test_image_names","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, to add TTA I did very little, in fact:\n- on each iteration, make 2 (or more) copies of the image.\n- have `ImageDataGenerator` augment them and then predict.\n\nBelow only changed parts:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_predictions(model):\n    ...\n\n    datagen = ImageDataGenerator(...)\n\n    for image_name in os.listdir(test_dir):\n        ...\n\n        images = np.broadcast_to(image, (N_FOLD,) + image.shape[1:])\n        cur_prediction = model.predict(datagen.flow(images))\n        cur_prediction = np.mean(cur_prediction, axis=0, keepdims=True)\n\n        ...\n\n    return predictions, test_image_names","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see, there's `broadcast_to` the number of folds for TTA, and then predict on them having `datagen` augment them.\n\nThis code, even though on each iterations there are only <= 10 copies of the image, started running into the memory error, which was not right."},{"metadata":{},"cell_type":"markdown","source":"I ran the experiment on my dev machine (without GPU :( ), and after a few hours I could see almost all of my RAM consumed, so I had a repro.\n\nHowever, when I started removing parts, I still had the same problem, both locally and on Kaggle, until I removed `model.predict`.\n\nSo, I started researching online, and apparently, not only I am facing the same problem:\n- [Repeatedly calling model.predict(...) results in memory leak](https://github.com/keras-team/keras/issues/13118)\n- [Memory leak on TF 2.0 with model.predict or/and model.fit with keras](https://github.com/tensorflow/tensorflow/issues/33030)\n- [Memory leak on TF 2.0 with model.predict or/and model.fit with keras](https://github.com/tensorflow/tensorflow/issues/34579)\n\nAnd even though some of the issues are closed, and the comments say there were a few fixes, evidently people are still hitting the issues in some cases.\n\nSo, the advice from there that worked for me was to use `predict_on_batch` (there's also `gc.collect` but it didn't work on its own):"},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_predictions(model):\n    ...\n\n    datagen = ImageDataGenerator(...)\n\n    for image_idx, image_name in enumerate(os.listdir(test_dir)):\n        ...\n\n        images = np.broadcast_to(image, (N_FOLD,) + image.shape[1:])\n        # Use predict_on_batch since looks like predict has a memory leak...\n        prediction = model.predict_on_batch(next(datagen.flow(images)))\n        ...\n\n        if image_idx % 100 == 0:  # Only run every so many images.\n            gc.collect()\n\n    return predictions, test_image_names","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}