{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Tensorflow ImageDataGenerator CNN\n# CNN Data Pipeline + Problem Solving Tutorial\n\nThis is the Kaggle version of the Github Codebase:\n- https://github.com/JamesMcGuigan/kaggle-bengali-ai\n\n\n# Bengali.AI Handwritten Grapheme Classification\n# Scores\n\n| Position              | Private | Public |  Prize       | Notes |\n|:----------------------|:-------:|:------:|-------------:|-------|\n| 1st Place             |  0.9762 | 0.9952 | \\$5000       |       |\n| 2nd Place             |  0.9689 | 0.9955 | \\$2000       |       |\n| 3rd Place             |  0.9645 | 0.9945 | \\$1000       |       |\n| Top 14                |  0.9491 | 0.9913 | Gold Medal   |       |\n| Top 5%  (102)         |  0.9348 | 0.9858 | Silver Medal |       |\n| Top 10% (205)         |  0.9306 | 0.9791 | Bronze Medal |       |\n| [ImageDataGenerator - CNN](https://www.kaggle.com/jamesmcguigan/bengali-ai-imagedatagenerator-cnn?scriptVersionId=31218596) |  0.9010 | 0.9413 | | 4\\*4 CNN + 1\\*256 Dense |\n| [ImageDataGenerator - CNN](https://www.kaggle.com/jamesmcguigan/bengali-ai-imagedatagenerator-cnn?scriptVersionId=31203616) |  0.8961 | 0.9482 | | 3\\*5 CNN + 1\\*256 Dense + Y+=grapheme |\n| [ImageDataGenerator - CNN](https://www.kaggle.com/jamesmcguigan/bengali-ai-imagedatagenerator-cnn?scriptVersionId=30636537) |  0.8921 | 0.9396 | | 3\\*4 CNN + 2\\*256 Dense + Y+=grapheme |\n| [Multi Output DF CNN](https://www.kaggle.com/jamesmcguigan/bengali-ai-multi-output-df-cnn?scriptVersionId=31204140)         |  0.8901 | 0.9402 | | 3\\*5 CNN + 1\\*256 Dense + Y+=grapheme - no augmentation |\n| [Multi Output DF CNN](https://www.kaggle.com/jamesmcguigan/bengali-ai-multi-output-df-cnn?scriptVersionId=30830488)         |  0.8828 | 0.9337 | | 3\\*4 CNN + 2\\*256 Dense + Y+=grapheme - no augmentation |\n| [ImageDataGenerator - CNN](https://www.kaggle.com/jamesmcguigan/bengali-ai-imagedatagenerator-cnn?scriptVersionId=31262979) |  0.8797 | 0.9198 | | 4\\*4 CNN + 1\\*256 Dense + Y+=grapheme + Regularization  |\n| sample_submission.csv |  0.0614 | 0.0614 | Random Score |       |\n\n\n\n# Dataset\nThe parquet training dataset consists of 200,840 grayscale images (in 4 files) at 137x236 resolution with a total\n filesize of 4.8GB of data.\n\n```\n./requirements.sh                                             # create ./venv/\n\nkaggle competitions download -c bengaliai-cv19 -p ./input/\nunzip ./input/bengaliai-cv19.zip -d ./input/bengaliai-cv19/\n\ntime ./src/preprocessing/write_images_to_filesystem.py        # optional\n```\n\n> Bengali is the 5th most spoken language in the world with hundreds of million of speakers. It’s the official\n> language of Bangladesh and the second most spoken language in India. Considering its reach, there’s significant\n> business and educational interest in developing AI that can optically recognize images of the language handwritten\n> . This challenge hopes to improve on approaches to Bengali recognition.\n>\n> Optical character recognition is particularly challenging for Bengali. While Bengali has 49 letters (to be more\n> specific 11 vowels and 38 consonants) in its alphabet, there are also 18 potential diacritics, or accents. This\n> means that there are many more graphemes, or the smallest units in a written language. The added complexity results\n> in ~13,000 different grapheme variations (compared to English’s 250 graphemic units).\n>\n> For this competition, you’re given the image of a handwritten Bengali grapheme and are challenged to separately\n> classify three constituent elements in the image: grapheme root, vowel diacritics, and consonant diacritics.\n>\n> This dataset contains images of individual hand-written \n> [Bengali characters](https://en.wikipedia.org/wiki/Bengali_alphabet). Bengali characters (graphemes) are written by\n> combining three components: a grapheme_root, vowel_diacritic, and consonant_diacritic. Your challenge is to classify \n> the components of the grapheme in each image. There are roughly 10,000 possible graphemes, of which roughly 1,000\n> are represented in the training set. The test set includes some graphemes that do not exist in train but has no\n> new grapheme components. It takes a lot of volunteers filling out \n> [sheets like this](https://github.com/BengaliAI/graphemePrepare/blob/master/collection/A4/form_1.jpg)\n> to generate a useful amount of real data; focusing the problem on the grapheme components rather than on recognizing\n> whole graphemes should make it possible to assemble a Bengali OCR system without handwriting samples for all 10,000\n> graphemes.\n\n      \n\n# Notebooks\n\n## Previous Competitions\n- [Kaggle Competition Entry - MNIST Digit Recognizer](https://github.com/JamesMcGuigan/kaggle-digit-recognizer)\n\n## Technical Research\n- [Reading Parquet Files RAM CPU Optimization](https://www.kaggle.com/jamesmcguigan/reading-parquet-files-ram-cpu-optimization)]\n- [Jupyter Environment Variable os.environ](https://www.kaggle.com/jamesmcguigan/jupyter-environment-variable-os-environ)]\n\n## Image Preprocessing\n- [Image Processing](https://www.kaggle.com/jamesmcguigan/bengali-ai-image-processing)\n- [Dataset as Image Directory](https://www.kaggle.com/jamesmcguigan/bengali-ai-dataset-as-image-directory)\n\n## Exploratory Data Analysis \n- [EDA Grapheme Combinations](https://www.kaggle.com/jamesmcguigan/bengali-ai-dataset-eda-grapheme-combinations)]\n- [Unicode Visualization of the Bengali Alphabet](https://www.kaggle.com/jamesmcguigan/unicode-visualization-of-the-bengali-alphabet)]\n\n## Writeup and Submission\n- [Bengali AI - CNN Data Pipeline + Problem Solving](https://www.kaggle.com/jamesmcguigan/bengali-ai-cnn-data-pipeline-problem-solving)\n\nThe Exploratory Data Analysis showed that only certain combinations of vowel/consonant diacritics where regularly\n combined with certain grapheme roots. \n \nVowel / Consonant Combinations:\n- Vowel #0 and Consonant #0 combine with everything\n- Vowels #3, #5, #6, #8 have limited combinations with Consonants\n- Consonant #3 is never combined except with Vowel #0\n- Consonant #6 only combineds with Vowels #0 and #1\n\nGrapheme Root Combinations:\n- Vowel #0 and Consonant #0 combine with (nearly) everything\n- ALL Roots combine with some Consonant #0\n- Several Roots do NOT combine with Vowel #0 = [26, 28, 33, 34, 73, 82, 108, 114, 126, 152, 157, 158, 163]\n- Several Roots do combine ALL Vowels = [13, 23, 64, 72, 79, 81, 96, 107, 113, 115, 133, 147]}\n- Only Root #107 combines with ALL Consonants\n\nIt was further discovered that Unicode itself is encoded as a multibyte string, using a lower level of base_graphemes \nthan root/vowel/consonant diacritics. Some Benglai Graphemes have multiple renderings for the same root/vowel/consonant \ncombination, which is implemented in unicode by allowing duplicate base_graphemes within the encoding.\n\nA `combination_matrix()` function was written that permitted a Unicode Visualization of the Bengali Language, \ntabulating the different combinations of grapheme_roots, vowels and consonants.\n- [src/jupyter/combination_matrix.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/jupyter/combination_matrix.py)\n\n\n# Technical Challenges\n\n# Kaggle Kernels Only Competition\n\nSolution: [kaggle_compile.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/kaggle_compile.py) \n \nThis codebase is approximately 2k CLOC, and builds upon my previous efforts for the \n[MINST Digit Recognizer](https://github.com/JamesMcGuigan/kaggle-digit-recognizer) competition.\n\nDeveloping a professionally engineered data pipeline using only a Jupyter Notebook would be impractical, \nand my preferred development workflow is to use the IntelliJ IDE and commit my code to github.\n\nMy first attempt involved manually copy and pasting required functions into a Kaggle Script Notebook, but then I\n decided that this process could be automated with a script.\n \n[./kaggle_compile.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/kaggle_compile.py) is a simple python script compiler. It reads in a python executable\n script, parses the `import` headers for any local include files, then recursively builds a dependency tree and \n concatenates these into single python file (much like an old-school javascript compiler). It can be called with\n  either `--save` or `--commit` cli flags to automattically save to disk or commit the result to git.\n  \nLimitations: the script only works for `from local.module import function` syntax, and does not support `import local\n.module` syntax, as calling `module.function()` inside a script would not work with script concatenation. Also the\nentire python file for the dependency is included, which does not guarantee the absence of namespace conflicts, but\nwith awareness of this issue good coding practices it is sufficiently practical for generating Kaggle submissions.\n  \nThere are other more robust solutions to this problem such as [stickytape](https://github.com/mwilliamson/stickytape), \nwhich allow for module imports, however the code for dependency files is obfuscated into a single line string\nvariable, which makes for an unreadable Kaggle submission. [./kaggle_compile.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/kaggle_compile.py) produces\nreadable and easily editable output.\n\n\n# Settings File\n\nAccording to the Kaggle Winning Model Documentation Guidelines \n- https://www.kaggle.com/WinningModelDocumentationGuidelines\n \n> settings.json\n> This file specifies the path to the train, test, model, and output directories. Here is an example file.\n> \n> This is the only place that specifies the path to these directories.\n> Any code that is doing I/O should use the appropriate base paths from SETTINGS.json\n\nThis was implemented as a python script [src/settings.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/settings.py) which defines default settings and\n hyperparameters.\n\n\n# Different Hyperparameter Configurations for Localhost, Kaggle Interactive and Kaggle Batch Mode\n\nI discovered that it is possible to detect which environment the python code is running in using\n```\nos.environ.get('KAGGLE_KERNEL_RUN_TYPE', 'Localhost')  # == 'Localhost' | 'Interactive' | 'Batch'\n```\n\nThis was explored in the following notebook:\n- [Jupyter Environment Variable os.environ](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/notebooks/Jupyter%20Environment%20Variable%20os.environ.ipynb)\n\nThe main uses for this are in the [settings.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/settings.py) file to specify different environment filesystem paths \n(eg `./input/bengaliai-cv19/` vs `../input/bengaliai-cv19/`) as well as different default timeout and verbosity\n settings. \n\nA second use is tweak hyperparameter settings in Interactive Mode (ie editing a script in Kaggle), such as setting \n`epochs=0` or `epochs=1` to allow quick sanity checking of the final script and that `submission\n.csv` is generated without throwing an exception. Using this flag removes the danger forgetting to revert these\nchanges and Batch Mode commit accidentally running using debug configuration.\n\n\n# Modifing Hyperparameters using CLI \n\nSolution: [src/util/argparse.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/util/argparse.py)\n\nMost of the hyperparameters in this codebase have been abstracted into dictionary arguments. As an alternative to\n editing the code, or using `KAGGLE_KERNEL_RUN_TYPE` it would be nice to be able to directly\n modify them in Localhost mode on the CLI.\n \n[src/util/argparse.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/util/argparse.py) provides two useful functions `argparse_from_dict({})` and \n`argparse_from_dicts([{},{}])` that will autogenerate argparse CLI config from a dictionary or list of dictionaries\n, with the option to edit them inplace. The result is that useful CLI flags can be implemented with a single line of\n code.\n \nThis is useful for passing `--timeout 1m` or `--epochs 0` when debugging locally. It can also be used as a basic\n method for hyperparameter search.\n\n\n# Out of Memory Errors\n\nEarly attempts at running this codebase as a Kaggle Script often terminated in `Exit code 137` (Out of Memory), and\nthat the 16GB of RAM available is insufficient to both load the entire training dataset into memory as well as a\ntensorflow model.\n\nThis prompted a full investigation into memory profiling and the different methods to read parquet files: \n- [Reading Parquet Files RAM CPU Optimization](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/notebooks/Reading%20Parquet%20Files%20RAM%20CPU%20Optimization.ipynb)\n \nThe dataformat of the parquet files means they cannot be read row-by-row, but only in their entirety. \n \nThe optimal method chosen was to using `pd.read_parquet()` but to place this inside a generator function. Each\nparquet would be read 3 times, but the generator would only store 1/3 of the raw parquet data in memory at any one\ntime. This reduced peak memory usage from `1.7GB` to `555MB` whilst increasing disk-IO timings from `46s` to `116s\n` (`+74s`) per epoch. \n\nThis was considered an acceptable (if overly conservative) RAM/Disk-IO tradeoff, but solved the Out of Memory errors.\n\nGenerator Source Code: \n- [src/dataset/ParquetImageDataGenerator.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/dataset/ParquetImageDataGenerator.py)\n\nAdditional memory optimizations involved using `float16` rather than `float64` during image normalization, and\n rescaling the image size by 50% before feeding it into the neural network model. \n\n\n# Notebook Exceeded Allowed Compute\n\nThis Kaggle competition has a `120m` maximum runtime constraint.\n\nThe simplest solution to this is to write a custom tensorflow callback that sets a timer and exits training before\nthe deadline. The source code for KaggleTimeoutCallback() is in: \n - [src/callbacks/KaggleTimeoutCallback.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/callbacks/KaggleTimeoutCallback.py)\n\n\n# Pretrained Model Weights\n\nAnother way around the timeout issue is to use pretrained model weights, run on Localhost without a timer, upload these\nfiles to Kaggle as a private dataset, and then Commit the kaggle script using `epochs=0`. \n\nThis requires using the `ModelCheckpoint()` callback to save the weights to a file, `glob2.glob(f\"../input\n/**/{os.path.basename(model_file)}\")` to find the relevant file within the attached Kaggle dataset, and \n`model.load_weights()` to load them again before training. \n\nThis was the method used to generate a score of 0.9396, which took 3h to train to convergence on a Localhost laptop.\n\n\n# Submission - Notebook Threw Exception\n\nThe public test dataset comprises only 12 test images, which is deceptively small. The result is that the Kaggle\n  Kernel can compile without exception, yet submitting the final `submission.csv` will generate a `Notebook Threw\n  Exception` error.\n  \nThe root cause is that the hidden test dataset is significantly larger, and presumably comparable in size to the\n   training dataset. This can cause both out-of-memory errors and timeout errors if the code for generating a \n  `submission.csv` file is not performance optimized.\n\nThere is known bug in `model.predict(...)` that results in a memory leak if the function is called in a loop. The\n  solution to this is to call `model.predict_on_batch(...)` which was combined with using a generator to read the test\n  parquet files, and a `for loop` to manually batch the data before calling `model.predict_on_batch(...)`  \n- https://github.com/keras-team/keras/issues/13118\n\nA second issue is that the `submission.csv` has a funny format, which is not a 4 column table, but rather in\n `{image_id}_{column}, {value}` format. My initial naive code implementation used a nested `for loop` to write the\n  values of `submission.csv` cell by cell. Rewriting this using a vectorized `df.apply()` solved the \n  `Notebook Threw Exception` error.\n\nOptimized code for generating submission.csv: \n- [src/util/csv.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/util/csv.py)\n\n\n# Model Generating Random Results\n\nThe random score of the `sample_submission.csv` is 0.0614, and several of my submission returned this result.\n\nDebugging this discovered several code issues:\n- Normalizing the data for training, but not for testing\n- Incorrect ordering of keys in `submission.csv` - https://www.kaggle.com/human-protein-atlas-image-classification/discussion/69366 \n- Training using `loss_weights=sqrt(output_shape.values())`\n- Using `tf.keras.backend.set_floatx('float16')` then retraining on pretrained model weights\n\n\n\n# Image Preprocessing\n\nThe following transforms where conducted as preprocessing:\n- Invert:  set background to 0 and text to 255\n- Resize:  reduce image size by 50% using `skimage.measure.block_reduce((1,2,2,1), np.max)`\n- Rescale: rescale brightness of maximum pixel to 255\n- Denoise: set background pixels less than maximum mean (42) to 0\n- Center:  crop image background border, and recenter image\n- Normalize: Convert pixel value range from `0 - 255` to `0.0 - 1.0`\n\nThis provided a clean and standarized format for both training and test data.\n\nAn exploration and testing of Image Preprocessing was conducted in Jupyter Notebook\n- [Image Processing](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/notebooks/Image%20Processing.ipynb)\n\nThe final code for image transforms\n- [src/dataset/Transforms.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/dataset/Transforms.py)\n\n\n# Image Augmentation\n`ImageDataGenerator` was used to provide Image Augmentation with the following settings:\n \n```\ndatagen_args = {\n    # \"rescale\":          1./255,  # \"normalize\": True is default in Transforms\n    # \"brightness_range\": 0.5,   # Preprocessing normalized brightness\n    \"zoom_range\":         0.2,\n    \"width_shift_range\":  0.1,     # Preprocessing already centered image\n    \"height_shift_range\": 0.1,     # Preprocessing already centered image\n    \"rotation_range\":     45/2,\n    \"shear_range\":        45/2,\n    \"fill_mode\":         'constant',\n    \"cval\": 0,\n    # \"featurewise_center\": True,             # No visible effect in plt.imgshow()\n    # \"samplewise_center\": True,              # No visible effect in plt.imgshow()\n    # \"featurewise_std_normalization\": True,  # No visible effect in plt.imgshow() | requires .fit()\n    # \"samplewise_std_normalization\": True,   # No visible effect in plt.imgshow() | requires .fit()\n    # \"zca_whitening\": True,                  # Kaggle, insufficent memory\n}\n```\n\nDue the memory constraints of not being able to load the entire training dataset into a single dataframe, \nand the performance issues of saving the test/train datasets to a directory tree of image files,\na ImageDataGenerator subclass was written with a custom `ParquetImageDataGenerator.flow_from_parquet()`  \nmethod. This used the generator pattern as described above to read from the parquet files, and run \npreprocessing `Transforms` on batches of data, whilst keep the minimum data possible in RAM.\n- [src/dataset/ParquetImageDataGenerator.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/dataset/ParquetImageDataGenerator.py)\n\n\n# Writing Images to Filesystem\n- [src/preprocessing/write_images_to_filesystem.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/preprocessing/write_images_to_filesystem.py)\n- [kaggle:jamesmcguigan/bengali-ai-dataset-as-image-directory](https://www.kaggle.com/jamesmcguigan/bengali-ai-dataset-as-image-directory)\n\nAs an experiment, a script was written to write the images in the parquet file to the filesystem after Image\n Preprocessing. \n\nThis took 11 minutes of runtime to complete on Kaggle, generating 200,840 image files.\n\nIn theory this would have allowed `ImageDataGenerator.flow_from_dictectory()` to be used, and this directory tree could\n have been imported as an external dataset. On localhost, having 200k files in a single directory tree produces\n performance issues, and this method was abandoned in favour of `ParquetImageDataGenerator.flow_from_parquet()`\n\n\n# Neural Network Models\n\n# Single Output CNN\n- [src/pipelines/simple_triple_df_cnn.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/pipelines/simple_triple_df_cnn.py)\n- [src/models/SingleOutputCNN.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/models/SingleOutputCNN.py)\n\nFirst attempt, simplest thing that could possibly work, was to port the CNN code used for MINST,\nload the data for each parquet file into a pandas DataFrame and train 3 separate neural networks, \none for each output variable.\n\nThe main issue with this simplistic approach is that the data must be read three times, and the entire training\n process takes three times as long.\n\nUnable to obtain a Kaggle Score using this method.\n \n\n# Multi Output CNN\n- [src/pipelines/multi_output_df_cnn.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/pipelines/multi_output_df_cnn.py)\n- [src/models/MultiOutputCNN.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/models/MultiOutputCNN.py)\n\nThe next evolution was to write a single CNN neural network with multiple outputs, thus reducing training time by 3x.\n\nThe model was trained using all 4 of the columns provided in the `train.csv`\n- grapheme_root\n- vowel_diacritic\n- consonant_diacritic\n- grapheme\n\n\nThe CNN Architecture uses multiple layers of:\n- Conv2D(3x3)\n- MaxPooling2D()\n- BatchNormalization()\n- Flatten()\n- Dropout()\n- Dense()\n- Dropout()\n\n\nTODO:\n- split the grapheme into constituent unicode bytes and retrain using softmax Multi-Hot-Encoding\n- Test to see if this method can generate a Kaggle score using only Image Preprocessing \n\n\n# ImageDataGenerator CNN\n- [src/pipelines/image_data_generator_cnn.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/pipelines/image_data_generator_cnn.py)\n- [src/models/MultiOutputCNN.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/models/MultiOutputCNN.py)\n- [kaggle:jamesmcguigan/bengali-ai-imagedatagenerator-cnn?scriptVersionId=30636537](https://www.kaggle.com/jamesmcguigan/bengali-ai-imagedatagenerator-cnn?scriptVersionId=30636537)\n- [data_output/scores/0.9396/image_data_generator_cnn-cnns_per_maxpool=3-maxpool_layers=4-dense_layers=2-dense_units=256-regularization=False-global_maxpool=False-submission.log](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/data_output/scores/0.9396/image_data_generator_cnn-cnns_per_maxpool=3-maxpool_layers=4-dense_layers=2-dense_units=256-regularization=False-global_maxpool=False-submission.log)\n\nThis approach used `ImageDataGenerator()` for Image Augmentation with a custom \n`ParquetImageDataGenerator.flow_from_parquet()` subclass function to read the parquet files in a memory optimized\n fashion\n\nThe following hyperparameters generated a 0.9396 Kaggle Score (44/2059 = top 3% | Late Submission) \n```\nmodel_hparams = {\n    \"cnns_per_maxpool\":   3,\n    \"maxpool_layers\":     4,\n    \"dense_layers\":       2,\n    \"dense_units\":      256,\n    \"regularization\": False,\n    \"global_maxpool\": False,\n}\ntrain_hparams = {\n    \"optimizer\":     \"RMSprop\",\n    \"scheduler\":     \"constant\",\n    \"learning_rate\": 0.001,\n    \"best_only\":     True,\n    \"batch_size\":    128,     \n    \"patience\":      10,\n    \"epochs\":        999,\n    \"loss_weights\":  False,\n}\n```\n\n`EarlyStopping(patience=10)` was used with a 75/25 train/test split (3 parquet files for train, 1 for test), and\n took 3.4 hours to converge on Localhost after 77 epochs\n\n\nIncreasing dense_units from `256` -> `512`:\n- converged in less epochs: `77` -> `66`\n- slightly quicker to train: `3.4h` -> `2.5h`\n- had slightly worse accuracy: `0.969` -> `0.9346` (`val_grapheme_root_accuracy`) \n\nTODO: \n- Run a full hyperparameter search to discover if it is possible to obtain a higher score\n\n\n# ImageDataGenerator Keras Application\n\nAn alternative to a custom designed CNN is to use a published CNN network architecture from `tf.keras.applications`\n\nThis requires removing the first and last layers of the application, then adding additional Dense layers before\n splitting the output.\n\nThe first application investigated was NASNetMobile, however compared to the custom CNN model: \n- converged in less epochs: `77` -> `53`\n- much slower to train: `3.4h` -> `4.15h`\n- had much worse accuracy: `0.969` -> `0.873` (`val_grapheme_root_accuracy`) \n\nTODO:\n- Test the full range of `tf.keras.applications`\n- Test if running for a fixed number of epochs produces more consistent results\n\n\n# Hyperparameter Search\n\nAdditional Visualizations of Hyperparameter Search\n- [notebooks/Hyperparamer Search.ipynb](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/notebooks/Hyperparamer Search.ipynb)\n\nPrevious research on MINST produced the following shortlist of Optimizer / Learning Rate / Scheduler combinations \n```\n\"optimized_scheduler\": {\n    \"Adagrad_triangular\": { \"learning_rate\": 0.1,    \"optimizer\": \"Adagrad\",  \"scheduler\": \"CyclicLR_triangular\"  },\n    \"Adagrad_plateau\":    { \"learning_rate\": 0.1,    \"optimizer\": \"Adagrad\",  \"scheduler\": \"plateau2\"      },\n    \"Adam_triangular2\":   { \"learning_rate\": 0.01,   \"optimizer\": \"Adam\",     \"scheduler\": \"CyclicLR_triangular2\" },\n    \"Nadam_plateau\":      { \"learning_rate\": 0.01,   \"optimizer\": \"Nadam\",    \"scheduler\": \"plateau_sqrt\"  },\n    \"Adadelta_plateau\":   { \"learning_rate\": 1.0,    \"optimizer\": \"Adadelta\", \"scheduler\": \"plateau10\"     },\n    \"SGD_triangular2\":    { \"learning_rate\": 1.0,    \"optimizer\": \"SGD\",      \"scheduler\": \"CyclicLR_triangular2\" },\n    \"RMSprop_constant\":   { \"learning_rate\": 0.001,  \"optimizer\": \"RMSprop\",  \"scheduler\": \"constant\"      },\n}\n```\n- https://github.com/JamesMcGuigan/kaggle-digit-recognizer\n\n# Optimizers and Learning Rates\n- [src/pipelines/image_data_generator_cnn_search.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/pipelines/image_data_generator_cnn_search.py)\n- [data_output/submissions/image_data_generator_cnn_search_train/results.csv](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/data_output/submissions/image_data_generator_cnn_search_train/results.csv)\n\n| Time | Epocs | Loss  | Optimizer | Scheduler | Learning Rate | Notes                    |\n|-----:|------:|:------|----------:|----------:|:--------------|--------------------------|\n| 7580 | 54\t   | 0.637 | Adadelta  | plateau10 | 1.000         | Lowest loss, least epochs, middle speed | \n| 9558 | 68\t   | 0.759 | Adam      | constant  | 0.001         | Lowest loss with constant optimizer | \n| 4514 | 32\t   | 0.818 | RMSProp   | constant  | 0.001         | Fastest with decent loss            | \n\n# CNN Model Hparams\n- [src/pipelines/image_data_generator_cnn_search.py](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/src/pipelines/image_data_generator_cnn_search.py)\n- [data_output/submissions/image_data_generator_cnn_search_model/results.csv](https://github.com/JamesMcGuigan/kaggle-bengali-ai/tree/master/data_output/submissions/image_data_generator_cnn_search_model/results.csv)\n\nA hyperparameter grid search was run varying the number of layers within the CNN model\n\nregularization\n- `regularization=False` almost always outperforms `regularization=True`\n- `regularization=True` prefers fewer dense units\n\nglobal_maxpool\n- `regularization=True` prefers `global_maxpool=False` (but not vice veras)\n- `global_maxpool=True` prefers double the number of `dense_units` and +1 `cnns_per_maxpool`\n\ncnns\n- increasing `maxpool_layers` prefers fewer `cnns_per_maxpool` (ideal total CNNs = 15 / 16) \n\ndense units\n- `dense_layers=1` is preferred over `2` or `3`\n\nFastest with high score \n- maxpool_layers=5 | cnns_per_maxpool=3 | dense_layers=1 | dense_units=256 | global_maxpool=False | regularization=False \n\nShortlist:\n- maxpool_layers=5 | cnns_per_maxpool=3 | dense_layers=1 | dense_units=512 | global_maxpool=True  | regularization=False\n- maxpool_layers=4 | cnns_per_maxpool=4 | dense_layers=1 | dense_units=256 | global_maxpool=False | regularization=False\n- maxpool_layers=4 | cnns_per_maxpool=4 | dense_layers=1 | dense_units=256 | global_maxpool=False | regularization=True\n\n\nResults:\n- Optimizing architecture from `3\\*4 CNN + 2\\*256 Dense` -> `3\\*5 CNN + 1\\*256 Dense` improves kaggle score `0.8921/0.9396` -> `0.8961/0.9482`\n- Adding regularization reduces score from `0.8961/0.9482` -> `0.8797/0.9198`\n- Removing `Y+=grapheme` improves private score from `0.8961/0.9482` -> `0.9010/0.9413`\n\nTODO:\n- Train Y on unicode grapheme components\n- Hyperparameter search on X_transforms + Augmentation params\n- Kaggle trained models still timeout, so pretrained models might produce better results  \n\n\n# Code","execution_count":null},{"metadata":{"_uuid":"44e2db4d-b88a-4531-bf4b-814017f602b0","_cell_guid":"d787dcfd-e6a4-46b3-9a33-dcb62c4ec0eb","trusted":true},"cell_type":"code","source":"#!/usr/bin/env python3\n\n##### \n##### ./kaggle_compile.py src/pipelines/image_data_generator_cnn.py --commit\n##### \n##### 2020-03-31 18:06:42+01:00\n##### \n##### origin\tgit@github.com:JamesMcGuigan/kaggle-bengali-ai.git (fetch)\n##### origin\tgit@github.com:JamesMcGuigan/kaggle-bengali-ai.git (push)\n##### \n##### * master b647190 [ahead 1] image_data_generator_cnn | Preserve test/train split for Kaggle\n##### \n##### b647190f8cbd70b31f472ea258be821ec0b3401d\n##### \n\n#####\n##### START src/settings.py\n#####\n\n# DOCS: https://www.kaggle.com/WinningModelDocumentationGuidelines\nimport os\n\nimport simplejson\n \nsettings = {}\n\nsettings['hparam_defaults'] = {\n\n    \"optimizer\":     \"RMSprop\",\n    \"scheduler\":     \"constant\",\n    \"learning_rate\": 0.001,\n    \"min_lr\":        0.001,\n    \"split\":         0.2,\n    \"batch_size\":    128,\n    \"fraction\":      1.0,\n    \"epochs\":         99,\n\n    \"patience\": {\n        'Localhost':    10,\n        'Interactive':  0,\n        'Batch':        10,\n    }.get(os.environ.get('KAGGLE_KERNEL_RUN_TYPE','Localhost'), 10),\n\n    \"loops\": {\n        'Localhost':   1,\n        'Interactive': 1,\n        'Batch':       1,\n    }.get(os.environ.get('KAGGLE_KERNEL_RUN_TYPE','Localhost'), 1),\n\n    # Timeout = 120 minutes | allow 30 minutes for testing submit | TODO: unsure of KAGGLE_KERNEL_RUN_TYPE on Submit\n    \"timeout\": {\n        'Localhost':   \"24h\",\n        'Interactive': \"5m\",\n        'Batch':       \"110m\",\n    }.get(os.environ.get('KAGGLE_KERNEL_RUN_TYPE','Localhost'), \"110m\")\n\n}\n\nsettings['verbose'] = {\n\n    \"tensorboard\": {\n        'Localhost':   True,\n        'Interactive': False,\n        'Batch':       False,\n    }.get(os.environ.get('KAGGLE_KERNEL_RUN_TYPE','Localhost'), False),\n\n    \"fit\": {\n        'Localhost':   1,\n        'Interactive': 2,\n        'Batch':       2,\n    }.get(os.environ.get('KAGGLE_KERNEL_RUN_TYPE','Localhost'), 2)\n\n}\n\nif os.environ.get('KAGGLE_KERNEL_RUN_TYPE'):\n    settings['dir'] = {\n        \"data\":        \"../input/bengaliai-cv19\",\n        \"features\":    \"./input_features/bengaliai-cv19/\",\n        \"models\":      \"./models\",\n        \"submissions\": \"./\",\n        \"logs\":        \"./logs\",\n    }\nelse:\n    settings['dir'] = {\n        \"data\":        \"./input/bengaliai-cv19\",\n        \"features\":    \"./input_features/bengaliai-cv19/\",\n        \"models\":      \"./data_output/models\",\n        \"submissions\": \"./data_output/submissions\",\n        \"logs\":        \"./logs\",\n    }\n\n####################\nif __name__ == '__main__':\n    for dirname in settings['dir'].values():\n        try:    os.makedirs(dirname, exist_ok=True)  # BUGFIX: read-only filesystem\n        except: pass\n    for key,value in settings.items():       print(f\"settings['{key}']:\".ljust(30), str(value))\n\n    if os.environ.get('KAGGLE_KERNEL_RUN_TYPE'):\n        with open('settings.json', 'w') as file:\n            print( 'settings', simplejson.dumps(settings, indent=4*' '))\n            simplejson.dump(settings, file, indent=4*' ')\n\n\n#####\n##### END   src/settings.py\n#####\n\n#####\n##### START src/dataset/Transforms.py\n#####\n\nimport math\nfrom typing import AnyStr, Dict, List, Union\n\nimport gc\nimport numpy as np\nimport pandas as pd\nimport skimage.measure\nfrom pandas import DataFrame, Series\n\n# from src.settings import settings\n\n\n\nclass Transforms():\n    csv_filename         = f\"{settings['dir']['data']}/train.csv\"\n    csv_data             = pd.read_csv(csv_filename).set_index('image_id', drop=True).astype('category')\n    csv_data['grapheme'] = csv_data['grapheme'].cat.codes.astype('category')\n\n\n    @classmethod\n    #@profile\n    def transform_Y(cls, df: DataFrame, Y_field: Union[List[str],str] = None) -> Union[DataFrame,Dict[AnyStr,DataFrame]]:\n        ### Profiler: 0.2% of DatasetDF() runtime\n        labels = df['image_id'].values\n        try:             output_df = cls.csv_data.loc[labels]\n        except KeyError: output_df = cls.csv_data.loc[[]]         # test dataset\n        output_df = output_df[Y_field] if Y_field else output_df\n\n        if isinstance(output_df, Series) or len(output_df.columns) == 1:\n            # single model output\n            output = pd.get_dummies( output_df )\n        else:\n            # multi model output\n            output = {\n                column: pd.get_dummies( output_df[column] )\n                for column in output_df.columns\n            }\n            del output['grapheme']\n        return output\n\n\n    # Source: https://www.kaggle.com/jamesmcguigan/bengali-ai-image-processing/\n    # noinspection PyArgumentList\n    @classmethod\n    #@profile\n    def transform_X(cls,\n                    train: DataFrame,\n                    resize=2,\n                    invert=True,\n                    rescale=True,\n                    denoise=True,\n                    center=True,\n                    normalize=True,\n    ) -> np.ndarray:\n        ### Profiler: 78.7% of DatasetDF() runtime\n        train = (train.drop(columns='image_id', errors='ignore')\n                 .values.astype('uint8')                   # unit8 for initial data processing\n                 .reshape(-1, 137, 236)                    # 2D arrays for inline image processing\n                )\n        # Colors   |   0 = black      | 255 = white\n        # invert   |   0 = background | 255 = line\n        # original | 255 = background |   0 = line\n\n        # Invert for processing\n        train = cls.invert(train)\n\n        if resize:\n            train = cls.resize(train, resize)\n\n        if rescale:\n            train = cls.rescale(train)\n\n        if denoise:\n            train = cls.denoise(train)\n\n        if center:\n            train = cls.center(train)\n\n        if not invert:\n            train = cls.invert(train)  # un-invert\n\n        if normalize:\n            train = cls.normalize(train)\n\n        train = train.reshape(*train.shape, 1)        # 4D ndarray for tensorflow CNN\n\n        gc.collect()  # ; sleep(1)\n        return train\n\n\n    @classmethod\n    #@profile\n    def invert(cls, train: np.ndarray) -> np.ndarray:\n        ### Profiler: 0.5% of DatasetDF() runtime\n        return (255-train)\n\n\n    @classmethod\n    #@profile\n    def normalize(cls, train: np.ndarray) -> np.ndarray:\n        ### Profiler: 15.4% of DatasetDF() runtime\n        train = train.astype('float16') / 255.0   # prevent division cast: int -> float64\n        return train\n\n\n    @classmethod\n    #@profile\n    def denoise(cls, train: np.ndarray) -> np.ndarray:\n        ### Profiler: 0.3% of DatasetDF() runtime\n        train = train * (train >= 42)  # 42 is the maximum mean\n        return train\n\n\n    @classmethod\n    #@profile\n    def rescale(cls, train: np.ndarray) -> np.ndarray:\n        ### Profiler: 3.4% of DatasetDF() runtime\n        ### Rescale lines to maximum brightness, and set background values (less than 2x mean()) to 0\n        ### max(mean()) =  14, 38,  33,  25, 36, 42,  20, 37,  38,  26, 36, 35\n        ### min(max())  = 242, 94, 105, 224, 87, 99, 247, 85, 106, 252, 85, 97\n        ### max(min())  =  0,   5,   3,   0,  6,  3,   0,  5,   3,   0,  6,  4\n        # try:\n        #     print('max mean()',  max([ np.mean(train[i]) for i in range(train.shape[0]) ]))\n        #     print('min  max()',  min([ np.max(train[i]) for i in range(train.shape[0]) ]))\n        #     print('max  min()',  max([ np.min(train[i]) for i in range(train.shape[0]) ]))\n        # except: pass\n        train = np.array([\n            (train[i].astype('float64') * 255./train[i].max()).astype('uint8')\n            for i in range(train.shape[0])\n        ])\n        return train\n\n\n    @classmethod\n    #@profile\n    def resize(cls, train: np.ndarray, resize: int) -> np.ndarray:\n        ### Profiler: 29% of DatasetDF() runtime  (37% with [for in] loop)\n        # NOTEBOOK: https://www.kaggle.com/jamesmcguigan/bengali-ai-image-processing/\n        # Out of the different resize functions:\n        # - np.mean(dtype=uint8) produces fragmented images (needs float16 to work properly - but RAM intensive)\n        # - np.median() produces the most accurate downsampling - but returns float64\n        # - np.max() produces an image with thicker lines       - occasionally produces bounding boxes\n        # - np.min() produces a  image with thiner  lines       - harder to read\n        if isinstance(resize, bool) and resize == True:\n            resize = 2\n        if resize and resize != 1:\n            resize_fn = np.max\n            if   len(train.shape) == 2: resize_shape =    (resize, resize)\n            elif len(train.shape) == 3: resize_shape = (1, resize, resize)\n            elif len(train.shape) == 4: resize_shape = (1, resize, resize, 1)\n            else:                       resize_shape = (1, resize, resize, 1)\n\n            train = skimage.measure.block_reduce(train, resize_shape, cval=0, func=resize_fn)\n            # train = np.array([\n            #     skimage.measure.block_reduce(train[i,:,:], (resize,resize), cval=0, func=resize_fn)\n            #     for i in range(train.shape[0])\n            # ])\n        return train\n\n\n    @classmethod\n    #@profile\n    def center(cls, train: np.ndarray) -> np.ndarray:\n        ### Profiler: 12.3% of DatasetDF() runtime\n        ### NOTE: cls.crop_center_image assumes inverted\n        train = np.array([\n            cls.crop_center_image(train[i,:,:], cval=0, tol=42)\n            for i in range(train.shape[0])\n        ])\n        return train\n\n\n    # DOCS: https://docs.scipy.org/doc/numpy/reference/generated/numpy.pad.html\n    # NOTE: assumes inverted\n    @classmethod\n    def crop_center_image(cls, img, cval=0, tol=0):\n        ### Profiler: 11% of DatasetDF() runtime\n        org_shape   = img.shape\n        img_cropped = cls.crop_image_border_px(img, px=1)\n        img_cropped = cls.crop_image_background(img_cropped, tol=tol)\n        pad_x       = (org_shape[0] - img_cropped.shape[0]) / 2.0\n        pad_y       = (org_shape[1] - img_cropped.shape[1]) / 2.0\n        padding     = (\n            (math.floor(pad_x), math.ceil(pad_x)),\n            (math.floor(pad_y), math.ceil(pad_y))\n        )\n        img_center = np.pad(img_cropped, padding, 'constant', constant_values=cval)\n        return img_center\n\n\n    @classmethod\n    def crop_image_border_px(cls, img, px=1):\n        ### Profiler: 0.1% of DatasetDF() runtime\n        ### crop one pixel border from the image to remove any bounding box effects\n        img  = img[px:img.shape[0]-px, px:img.shape[1]-px]\n        return img\n\n\n    # Source: https://codereview.stackexchange.com/questions/132914/crop-black-border-of-image-using-numpy\n    # This is the fast method that simply remove all empty rows/columns\n    # NOTE: assumes inverted\n    @classmethod\n    def crop_image_background(cls, img, tol=0):\n        ### Profiler: 4.7% of DatasetDF() runtime\n        img  = img[1:img.shape[0]-1, 1:img.shape[1]-1]  # crop one pixel border from image to remove any bounding box\n        mask = img > tol\n        return img[np.ix_(mask.any(1),mask.any(0))]\n\n\n#####\n##### END   src/dataset/Transforms.py\n#####\n\n#####\n##### START src/callbacks/KaggleTimeoutCallback.py\n#####\n\nimport math\nimport re\nimport time\nfrom typing import Union\n\nimport tensorflow as tf\n\n\nclass KaggleTimeoutCallback(tf.keras.callbacks.Callback):\n    start_python = time.time()\n\n\n    def __init__(self, timeout: Union[int, float, str], from_now=False, verbose=False):\n        super().__init__()\n        self.verbose           = verbose\n        self.from_now          = from_now\n        self.start_time        = self.start_python if not self.from_now else time.time()\n        self.timeout_seconds   = self.parse_seconds(timeout)\n\n        self.last_epoch_start  = time.time()\n        self.last_epoch_end    = time.time()\n        self.last_epoch_time   = self.last_epoch_end - self.last_epoch_start\n        self.current_runtime   = self.last_epoch_end - self.start_time\n\n\n    def on_train_begin(self, logs=None):\n        self.check_timeout()  # timeout before first epoch if model.fit() is called again\n\n\n    def on_epoch_begin(self, epoch, logs=None):\n        self.last_epoch_start = time.time()\n\n\n    def on_epoch_end(self, epoch, logs=None):\n        self.last_epoch_end  = time.time()\n        self.last_epoch_time = self.last_epoch_end - self.last_epoch_start\n        self.check_timeout()\n\n\n    def check_timeout(self):\n        self.current_runtime = self.last_epoch_end - self.start_time\n        if self.verbose:\n            print(f'\\nKaggleTimeoutCallback({self.format(self.timeout_seconds)}) runtime {self.format(self.current_runtime)}')\n\n        # Give timeout leeway of 2 * last_epoch_time\n        if (self.current_runtime + self.last_epoch_time*2) >= self.timeout_seconds:\n            print(f\"\\nKaggleTimeoutCallback({self.format(self.timeout_seconds)}) stopped after {self.format(self.current_runtime)}\")\n            self.model.stop_training = True\n\n\n    @staticmethod\n    def parse_seconds(timeout) -> int:\n        if isinstance(timeout, (float,int)): return int(timeout)\n        seconds = 0\n        for (number, unit) in re.findall(r\"(\\d+(?:\\.\\d+)?)\\s*([dhms])?\", str(timeout)):\n            if   unit == 'd': seconds += float(number) * 60 * 60 * 24\n            elif unit == 'h': seconds += float(number) * 60 * 60\n            elif unit == 'm': seconds += float(number) * 60\n            else:             seconds += float(number)\n        return int(seconds)\n\n\n    @staticmethod\n    def format(seconds: Union[int,float]) -> str:\n        runtime = {\n            \"d\":   math.floor(seconds / (60*60*24) ),\n            \"h\":   math.floor(seconds / (60*60)    ) % 24,\n            \"m\":   math.floor(seconds / (60)       ) % 60,\n            \"s\":   math.floor(seconds              ) % 60,\n        }\n        return \" \".join([ f\"{runtime[unit]}{unit}\" for unit in [\"h\", \"m\", \"s\"] if runtime[unit] != 0 ])\n\n\n#####\n##### END   src/callbacks/KaggleTimeoutCallback.py\n#####\n\n#####\n##### START src/dataset/DatasetDF.py\n#####\n\nimport gc\nimport os\nfrom typing import AnyStr, Dict, Union\n\nimport glob2\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\n\n# from src.dataset.Transforms import Transforms\n# from src.settings import settings\n\n\nclass DatasetDF():\n    csv_data = Transforms.csv_data\n\n\n    #@profile\n    def __init__(self,\n                 test_train   = 'train',\n                 data_id: Union[str,int] = '0',\n                 size         = -1,\n                 fraction     =  1,\n                 split: float = 0.1,\n                 Y_field      = None,\n                 shuffle      = True,\n                 transform_X_args = {},\n                 transform_Y_args = {},\n        ):\n        gc.collect()\n\n        self.test_train = test_train\n        self.data_id    = data_id\n        self.Y_field    = Y_field\n        self.split      = split    if self.test_train is 'train' else 0\n        self.shuffle    = shuffle  if self.test_train is 'train' else False\n        self.fraction   = fraction if self.test_train is 'train' else 1\n        self.size       = size\n\n        self.parquet_filenames = sorted(glob2.glob(f\"{settings['dir']['data']}/{test_train}_image_data_{data_id}.parquet\"))\n\n        self.X:  Dict[AnyStr, np.ndarray]               = { \"train\": np.ndarray((0,)), \"valid\": np.ndarray((0,)) }\n        self.Y:  Dict[AnyStr, Union[pd.DataFrame,Dict]] = { \"train\": pd.DataFrame(),   \"valid\": pd.DataFrame()   }\n        self.ID: Dict[AnyStr, np.ndarray]               = { \"train\": np.ndarray((0,)), \"valid\": np.ndarray((0,)) }\n        for parquet_filename in self.parquet_filenames:\n            raw = {\n                'train': pd.read_parquet(parquet_filename),\n                'valid': None\n            }\n            # Use size=1 to create a reference dataframe with valid .input_size() + .output_size()\n            if self.size > 0:\n                raw['valid'] = raw['train'][size+1:size*2]\n                raw['train'] = raw['train'][:size]\n            else:\n                if self.fraction < 1:\n                    raw['train'], discard      = train_test_split(raw['train'], train_size=self.fraction, shuffle=self.shuffle)\n                    del discard\n                if self.split != 0:\n                    raw['train'], raw['valid'] = train_test_split(raw['train'], test_size=self.split,     shuffle=self.shuffle, random_state=0)\n\n            if raw['valid'] is None:\n                raw['valid'] = pd.DataFrame(columns=raw['train'].columns)\n\n            # Attempt to save memory by doing transform_X() within the loop\n            # X can be transformed before np.concatenate, but multi-output Y must be done after pd.concat()\n            for key, value in raw.items():\n                X = Transforms.transform_X(value, **transform_X_args)\n                if len(self.X[key]) == 0: self.X[key] = X\n                else:                     self.X[key] = np.concatenate([ self.X[key], X ])\n                self.Y[key]  = pd.concat([      self.Y[key],  value[['image_id']]       ])\n                self.ID[key] = np.concatenate([ self.ID[key], value['image_id'].values  ])\n            del raw; gc.collect()\n\n        self.Y = {\n            key: Transforms.transform_Y(value, **transform_Y_args)\n            for key,value in self.Y.items()\n        }\n        pass\n\n\n    def epoch_size(self):\n        return self.X['train'].shape[0]\n\n\n    def input_shape(self):\n        return self.X['train'].shape[1:]  # == (137, 236, 1) / 2\n\n\n    @classmethod\n    def output_shape(cls, Y_field=None):\n        if isinstance(Y_field, str):\n            return cls.csv_data[Y_field].nunique()\n\n        csv_data     = cls.csv_data[Y_field] if Y_field else cls.csv_data\n        output_shape = (csv_data.drop(columns='image_id', errors='ignore')\n                                .drop(columns='grapheme', errors='ignore')                        \n                                .nunique()\n                                .to_dict())\n        print('output_shape',output_shape)\n        return output_shape\n\n\n\n\nif __name__ == '__main__' and not os.environ.get('KAGGLE_KERNEL_RUN_TYPE'):\n    ### NOTE: loading all datasets at once exceeds 12GB RAM and crashes Python (on 16GB RAM machine)\n    ### Runtime: 3m 12s - for in range(0,4)\n    ### $ find ./src/ -name '*.py' | xargs perl -p -i -e 's/#@profile/@profile/'\n    ### $ time python3 -m memory_profiler src/dataset/DatasetDF.py | less\n    #@profile()\n    def main():\n        for data_id in range(0,4):\n            for test_train in ['test', 'train']:\n                dataset = DatasetDF(test_train=test_train, data_id=data_id, fraction=1)\n                Y_shape = {}\n                for key, Y in dataset.Y.items():\n                    if isinstance(Y, dict): Y_shape[key] = { k:v.shape for k,v in Y.items() }\n                    else:                   Y_shape[key] = Y.shape\n\n                print(f\"{test_train}:{data_id} dataset.image_filenames\", dataset.parquet_filenames)\n                print(f\"{test_train}:{data_id} dataset.X\",               { key: df.shape for key, df in dataset.X.items() })\n                print(f\"{test_train}:{data_id} dataset.Y\", Y_shape)\n                print(f\"{test_train}:{data_id} dataset.input_shape()\",   dataset.input_shape())\n                print(f\"{test_train}:{data_id} dataset.output_shape()\",  dataset.output_shape())\n                print(f\"{test_train}:{data_id} dataset.epoch_size()\",    dataset.epoch_size())\n    main()\n\n#####\n##### END   src/dataset/DatasetDF.py\n#####\n\n#####\n##### START src/util/logs.py\n#####\n\nimport os\nimport time\nfrom datetime import datetime\nfrom typing import Dict, Union\n\nimport humanize\nimport simplejson\n\n# from src.settings import settings\n\n\n\ndef model_stats_from_history(history, timer_seconds=0, best_only=False) -> Union[None, Dict]:\n    if 'val_loss' in history.history:\n        best_epoch            = history.history['val_loss'].index(min( history.history['val_loss'] )) if best_only else -1\n        model_stats           = { key: value[best_epoch] for key, value in history.history.items() }\n        model_stats['time']   = timer_seconds\n        model_stats['epochs'] = len(history.history['loss'])\n    else:\n        model_stats = None\n    return model_stats\n\n\npython_start = time.time()\ndef log_model_stats(model_stats, logfilename, model_hparams, train_hparams):\n    os.makedirs(os.path.dirname(logfilename), exist_ok=True)\n    with open(logfilename, 'w') as file:\n        output = [\n            \"------------------------------\",\n            f\"Completed\",\n            \"------------------------------\",\n            f\"model_hparams: {model_hparams}\",\n            f\"train_hparams: {train_hparams}\",\n            \"------------------------------\",\n        ]\n        output += [ f\"settings[{key}]: {value}\" for key, value in settings.items() ]\n        output.append(\"------------------------------\")\n\n        if isinstance(model_stats, dict):\n            output.append(simplejson.dumps(\n                { key: str(value) for key, value in model_stats.items() },\n                sort_keys=False, indent=4*' '\n            ))\n        elif isinstance(model_stats, list):\n            output += list(map(str, model_stats))\n        else:\n            output.append( str(model_stats) )\n\n        output.append(\"------------------------------\")\n        output += [\n            f\"------------------------------\",\n            f\"script started: { datetime.fromtimestamp( python_start ).strftime('%Y-%m-%d %H:%M:%S')}\",\n            f\"script ended:   { datetime.fromtimestamp( time.time()  ).strftime('%Y-%m-%d %H:%M:%S')}\",\n            f\"script runtime: { humanize.naturaldelta(  python_start - time.time() )}s\",\n            f\"------------------------------\",\n        ]\n        output = \"\\n\".join(output)\n        print(      output )\n        file.write( output )\n        print(\"wrote:\", logfilename)\n\n\n#####\n##### END   src/util/logs.py\n#####\n\n#####\n##### START src/vendor/CLR/clr_callback.py\n#####\n\nfrom tensorflow.keras.callbacks import *\nimport tensorflow.keras.backend as K\nimport numpy as np\n\nclass CyclicLR(Callback):\n    \"\"\"This callback implements a cyclical learning rate policy (CLR).\n    The method cycles the learning rate between two boundaries with\n    some constant frequency, as detailed in this paper (https://arxiv.org/abs/1506.01186).\n    The amplitude of the cycle can be scaled on a per-iteration or \n    per-cycle basis.\n    This class has three built-in policies, as put forth in the paper.\n    \"triangular\":\n        A basic triangular cycle w/ no amplitude scaling.\n    \"triangular2\":\n        A basic triangular cycle that scales initial amplitude by half each cycle.\n    \"exp_range\":\n        A cycle that scales initial amplitude by gamma**(cycle iterations) at each \n        cycle iteration.\n    For more detail, please see paper.\n    \n    # Example\n        ```python\n            clr = CyclicLR(base_lr=0.001, max_lr=0.006,\n                                step_size=2000., mode='triangular')\n            model.fit(X_train, Y_train, callbacks=[clr])\n        ```\n    \n    Class also supports custom scaling functions:\n        ```python\n            clr_fn = lambda x: 0.5*(1+np.sin(x*np.pi/2.))\n            clr = CyclicLR(base_lr=0.001, max_lr=0.006,\n                                step_size=2000., scale_fn=clr_fn,\n                                scale_mode='cycle')\n            model.fit(X_train, Y_train, callbacks=[clr])\n        ```    \n    # Arguments\n        base_lr: initial learning rate which is the\n            lower boundary in the cycle.\n        max_lr: upper boundary in the cycle. Functionally,\n            it defines the cycle amplitude (max_lr - base_lr).\n            The lr at any cycle is the sum of base_lr\n            and some scaling of the amplitude; therefore \n            max_lr may not actually be reached depending on\n            scaling function.\n        step_size: number of training iterations per\n            half cycle. Authors suggest setting step_size\n            2-8 x training iterations in epoch.\n        mode: one of {triangular, triangular2, exp_range}.\n            Default 'triangular'.\n            Values correspond to policies detailed above.\n            If scale_fn is not None, this argument is ignored.\n        gamma: constant in 'exp_range' scaling function:\n            gamma**(cycle iterations)\n        scale_fn: Custom scaling policy defined by a single\n            argument lambda function, where \n            0 <= scale_fn(x) <= 1 for all x >= 0.\n            mode paramater is ignored \n        scale_mode: {'cycle', 'iterations'}.\n            Defines whether scale_fn is evaluated on \n            cycle number or cycle iterations (training\n            iterations since start of cycle). Default is 'cycle'.\n    \"\"\"\n\n    def __init__(self, base_lr=0.001, max_lr=0.006, step_size=2000., mode='triangular',\n                 gamma=1., scale_fn=None, scale_mode='cycle'):\n        super(CyclicLR, self).__init__()\n\n        self.base_lr = base_lr\n        self.max_lr = max_lr\n        self.step_size = step_size\n        self.mode = mode\n        self.gamma = gamma\n        if scale_fn == None:\n            if self.mode == 'triangular':\n                self.scale_fn = lambda x: 1.\n                self.scale_mode = 'cycle'\n            elif self.mode == 'triangular2':\n                self.scale_fn = lambda x: 1/(2.**(x-1))\n                self.scale_mode = 'cycle'\n            elif self.mode == 'exp_range':\n                self.scale_fn = lambda x: gamma**(x)\n                self.scale_mode = 'iterations'\n        else:\n            self.scale_fn = scale_fn\n            self.scale_mode = scale_mode\n        self.clr_iterations = 0.\n        self.trn_iterations = 0.\n        self.history = {}\n\n        self._reset()\n\n    def _reset(self, new_base_lr=None, new_max_lr=None,\n               new_step_size=None):\n        \"\"\"Resets cycle iterations.\n        Optional boundary/step size adjustment.\n        \"\"\"\n        if new_base_lr != None:\n            self.base_lr = new_base_lr\n        if new_max_lr != None:\n            self.max_lr = new_max_lr\n        if new_step_size != None:\n            self.step_size = new_step_size\n        self.clr_iterations = 0.\n        \n    def clr(self):\n        cycle = np.floor(1+self.clr_iterations/(2*self.step_size))\n        x = np.abs(self.clr_iterations/self.step_size - 2*cycle + 1)\n        if self.scale_mode == 'cycle':\n            return self.base_lr + (self.max_lr-self.base_lr)*np.maximum(0, (1-x))*self.scale_fn(cycle)\n        else:\n            return self.base_lr + (self.max_lr-self.base_lr)*np.maximum(0, (1-x))*self.scale_fn(self.clr_iterations)\n        \n    def on_train_begin(self, logs={}):\n        logs = logs or {}\n\n        if self.clr_iterations == 0:\n            K.set_value(self.model.optimizer.lr, self.base_lr)\n        else:\n            K.set_value(self.model.optimizer.lr, self.clr())        \n            \n    def on_batch_end(self, epoch, logs=None):\n        \n        logs = logs or {}\n        self.trn_iterations += 1\n        self.clr_iterations += 1\n\n        self.history.setdefault('lr', []).append(K.get_value(self.model.optimizer.lr))\n        self.history.setdefault('iterations', []).append(self.trn_iterations)\n\n        for k, v in logs.items():\n            self.history.setdefault(k, []).append(v)\n        \n        K.set_value(self.model.optimizer.lr, self.clr())\n\n\n#####\n##### END   src/vendor/CLR/clr_callback.py\n#####\n\n#####\n##### START src/dataset/ParquetImageDataGenerator.py\n#####\n\n# Notebook: https://www.kaggle.com/jamesmcguigan/reading-parquet-files-ram-cpu-optimization/\n# Notebook: https://www.kaggle.com/jamesmcguigan/bengali-ai-image-processing\nimport gc\nimport math\nfrom collections import Callable\n\nimport glob2\nimport pandas as pd\nfrom keras_preprocessing.image import ImageDataGenerator\nfrom pyarrow.parquet import ParquetFile\n\n\nclass ParquetImageDataGenerator(ImageDataGenerator):\n\n    def __init__(self, **kwargs):\n        super().__init__(**kwargs)\n\n\n    def flow_from_parquet(\n            self,\n            glob_path:       str,\n            transform_X:     Callable,\n            transform_Y:     Callable,\n            transform_X_args = {},\n            transform_Y_args = {},\n            batch_size       = 32,\n            reads_per_file   = 2,\n            resamples        = 1,\n            shuffle          = False,\n            infinite         = True,\n            test             = False,\n    ):\n        \"\"\"\n            Source: ./venv/lib/python3.6/site-packages/keras_preprocessing/image/image_data_generator.py\n            # Returns\n            An `Iterator` yielding tuples of `(x, y)`\n                where `x` is a numpy array of image data\n                (in the case of a single image input) or a list\n                of numpy arrays (in the case with\n                additional inputs) and `y` is a numpy array\n                of corresponding labels. If 'sample_weight' is not None,\n                the yielded tuples are of the form `(x, y, sample_weight)`.\n                If `y` is None, only the numpy array `x` is returned.\n        \"\"\"\n        if test:\n            shuffle  = False\n            infinite = False\n\n        for (X,Y) in self.cache_XY_generator(\n                glob_path=glob_path,\n                transform_X=transform_X,\n                transform_X_args=transform_X_args,\n                transform_Y=transform_Y,\n                transform_Y_args=transform_Y_args,\n                reads_per_file=reads_per_file,\n                resamples=resamples,\n                shuffle=shuffle,\n                infinite=infinite,\n        ):\n            cache_size  = X.shape[0]\n            batch_count = math.ceil( cache_size / batch_size )\n            for n_batch in range(batch_count):\n                X_batch = X[ batch_size * n_batch : batch_size * (n_batch+1) ].copy()\n                if isinstance(Y, dict):\n                    Y_batch = {\n                        key: Y[key][ batch_size * n_batch : batch_size * (n_batch+1) ].copy()\n                        for key in Y.keys()\n                    }\n                else:\n                    Y_batch = Y[ batch_size * n_batch : batch_size * (n_batch+1) ].copy()\n                yield ( X_batch, Y_batch )\n\n\n    @classmethod\n    def cache_XY_generator(\n            cls,\n            glob_path:        str,\n            transform_X:      Callable,\n            transform_X_args: {},\n            transform_Y:      Callable,\n            transform_Y_args: {},\n            reads_per_file  = 3,\n            resamples       = 1,\n            shuffle         = False,\n            infinite        = False,\n    ):\n        for cache in cls.cache_generator(\n                glob_path=glob_path,\n                reads_per_file=reads_per_file,\n                resamples=resamples,\n                shuffle=shuffle,\n                infinite=infinite,\n        ):\n            X = transform_X(cache, **transform_X_args)\n            Y = transform_Y(cache, **transform_Y_args)\n            yield (X, Y)\n\n\n    @classmethod\n    def cache_generator(\n            cls,\n            glob_path,\n            reads_per_file = 3,\n            resamples      = 1,\n            shuffle        = False,\n            infinite       = False,\n    ):\n        filenames = sorted(glob2.glob(glob_path))\n        if len(filenames) == 0: raise Exception(f\"{cls.__name__}.batch_generator() - invalid glob_path: {glob_path}\")\n\n        gc.collect();  # sleep(1)   # sleep(1) is required to allow measurement of the garbage collector\n        while True:\n            for filename in filenames:\n                num_rows    = ParquetFile(filename).metadata.num_rows\n                cache_size  = math.ceil( num_rows / reads_per_file )\n                for n_read in range(reads_per_file):\n                    gc.collect();  # sleep(1)   # sleep(1) is required to allow measurement of the garbage collector\n                    cache = (\n                        pd.read_parquet(filename)\n                            # .set_index('image_id', drop=True)  # WARN: Don't do this, it breaks other things\n                            .iloc[ cache_size * n_read : cache_size * (n_read+1) ]\n                            .copy()\n                    )\n                    for resample in range(resamples):\n                        if shuffle:\n                            cache = cache.sample(frac=1)\n                        yield cache\n            if not infinite: break\n\n#####\n##### END   src/dataset/ParquetImageDataGenerator.py\n#####\n\n#####\n##### START src/models/MultiOutputCNN.py\n#####\n\nimport inspect\nimport types\nfrom typing import Dict, List, Union, cast\n\nfrom tensorflow.keras import Input, Model, regularizers\nfrom tensorflow.keras.layers import (\n    BatchNormalization,\n    Conv2D,\n    Dense,\n    Dropout,\n    Flatten,\n    GlobalMaxPooling2D,\n    MaxPooling2D,\n)\n\n\n# noinspection DuplicatedCode\ndef MultiOutputCNN(\n        input_shape,\n        output_shape: Union[List, Dict],\n        cnns_per_maxpool=1,\n        maxpool_layers=1,\n        dense_layers=1,\n        dense_units=64,\n        dropout=0.25,\n        regularization=False,\n        global_maxpool=False,\n        name='',\n)  -> Model:\n    function_name = cast(types.FrameType, inspect.currentframe()).f_code.co_name\n    model_name    = f\"{function_name}-{name}\" if name else function_name\n    # model_name  = seq([ function_name, name ]).filter(lambda x: x).make_string(\"-\")  # remove dependency on pyfunctional - not in Kaggle repo without internet\n\n    inputs = Input(shape=input_shape)\n    x      = inputs\n\n    for cnn1 in range(1,maxpool_layers+1):\n        for cnn2 in range(1, cnns_per_maxpool+1):\n            x = Conv2D( 32 * cnn1, kernel_size=(3, 3), padding='same', activation='relu')(x)\n        x = MaxPooling2D(pool_size=(2, 2))(x)\n        x = BatchNormalization()(x)\n        x = Dropout(dropout)(x)\n\n    if global_maxpool:\n        x = GlobalMaxPooling2D()(x)\n\n    x = Flatten()(x)\n\n    for nn1 in range(0,dense_layers):\n        if regularization:\n            x = Dense(dense_units, activation='relu',\n                      kernel_regularizer=regularizers.l2(0.01),\n                      activity_regularizer=regularizers.l1(0.01))(x)\n        else:\n            x = Dense(dense_units, activation='relu')(x)\n\n        x = BatchNormalization()(x)\n        x = Dropout(dropout)(x)\n\n    x = Flatten(name='output')(x)\n\n    if isinstance(output_shape, dict):\n        outputs = [\n            Dense(output_shape, activation='softmax', name=key)(x)\n            for key, output_shape in output_shape.items()\n        ]\n    else:\n        outputs = [\n            Dense(output_shape, activation='softmax', name=f'output_{index}')(x)\n            for index, output_shape in enumerate(output_shape)\n        ]\n\n    model = Model(inputs, outputs, name=model_name)\n    # plot_model(model, to_file=os.path.join(os.path.dirname(__file__), f\"{name}.png\"))\n    return model\n\n\n#####\n##### END   src/models/MultiOutputCNN.py\n#####\n\n#####\n##### START src/util/argparse.py\n#####\n\nimport argparse\nimport copy\nfrom typing import Dict, List\n\n\n\ndef argparse_from_dicts(configs: List[Dict], inplace=False) -> List[Dict]:\n    parser = argparse.ArgumentParser()\n    for config in list(configs):\n        for key, value in config.items():\n            if isinstance(value, bool):\n                parser.add_argument(f'--{key}', action='store_true', default=value, help=f'{key} (default: %(default)s)')\n            else:\n                parser.add_argument(f'--{key}', type=type(value),    default=value, help=f'{key} (default: %(default)s)')\n\n\n    args, unknown = parser.parse_known_args()  # Ignore extra CLI args passed in by Kaggle\n\n    outputs = configs if inplace else copy.deepcopy(configs)\n    for index, output in enumerate(outputs):\n        for key, value in outputs[index].items():\n            outputs[index][key] = getattr(args, key)\n\n    return outputs\n\n\ndef argparse_from_dict(config: Dict, inplace=False):\n    return argparse_from_dicts([config], inplace)[0]\n\n\n#####\n##### END   src/util/argparse.py\n#####\n\n#####\n##### START src/util/csv.py\n#####\n\nimport os\nfrom itertools import chain\n\nimport gc\nimport numpy as np\nimport pandas as pd\nfrom pandas import DataFrame\n\n# from src.dataset.DatasetDF import DatasetDF\n# from src.dataset.ParquetImageDataGenerator import ParquetImageDataGenerator\n# from src.dataset.Transforms import Transforms\n# from src.settings import settings\n\n\n\n### BUGFIX: Repeatedly calling model.predict(...) results in memory leak - https://github.com/keras-team/keras/issues/13118\ndef submission_df(model, output_shape):\n    gc.collect()\n\n    submission = pd.DataFrame(columns=output_shape.keys())\n    # large datasets on submit, so loop\n    for data_id in range(0,4):\n        test_dataset      = DatasetDF(test_train='test', data_id=data_id, transform_X_args = {} )  # \"normalize\": True is default\n        test_dataset_rows = test_dataset.X['train'].shape[0]\n        batch_size        = 64\n        for index in range(0, test_dataset_rows, batch_size):\n            try:\n                X_batch     = test_dataset.X['train'][index : index+batch_size]\n                predictions = model.predict_on_batch(X_batch)\n                # noinspection PyTypeChecker\n                submission = submission.append(\n                    pd.DataFrame({\n                        key: np.argmax( predictions[index], axis=-1 )\n                        for index, key in enumerate(output_shape.keys())\n                        }, index=test_dataset.ID['train'])\n                    )\n            except Exception as exception:\n                print('submission_df_generator()', exception)\n\n        return submission\n\n###\n### Use submission_df() it seems to have more success on Kaggle\n###\ndef submission_df_generator(model, output_shape):\n    gc.collect()\n\n    # if os.environ.get('KAGGLE_KERNEL_RUN_TYPE', 'Interactive') == 'Interactive':\n    if os.environ.get('KAGGLE_KERNEL_RUN_TYPE') == 'Interactive':\n        globpath = f\"{settings['dir']['data']}/train_image_data_*.parquet\"\n    else:\n        globpath = f\"{settings['dir']['data']}/test_image_data_*.parquet\"\n\n    # large datasets on submit, so loop via generator to avoid Out-Of-Memory errors\n    submission  = pd.DataFrame(columns=output_shape.keys())\n    cache_index = 0\n    for cache in ParquetImageDataGenerator.cache_generator(\n            globpath,\n            reads_per_file = 2,\n            resamples      = 1,\n            shuffle        = False,\n            infinite       = False,\n    ):\n        try:\n            cache_index      += 1\n            batch_size        = 128\n            test_dataset_rows = cache.shape[0]\n            print(f'submission_df_generator() - submission: ', cache_index, submission.shape)\n            if test_dataset_rows == 0: continue\n            for index in range(0, test_dataset_rows, batch_size):\n                try:\n                    batch = cache[index : index+batch_size]\n                    if batch.shape[0] == 0: continue\n                    X           = Transforms.transform_X(batch)  # normalize=True is default\n                    predictions = model.predict_on_batch(X)\n                    submission  = submission.append(\n                        pd.DataFrame({\n                            key: np.argmax( predictions[index], axis=-1 )\n                            for index, key in enumerate(output_shape.keys())\n                        }, index=batch['image_id'])\n                    )\n                except Exception as exception:\n                    print('submission_df_generator() - batch', exception)\n        except Exception as exception:\n            print('submission_df_generator() - cache', exception)\n\n    return submission\n\n\n\n# def df_to_submission(df: DataFrame) -> DataFrame:\n#     print('df_to_submission() - input', df.shape)\n#     output_fields = ['consonant_diacritic', 'grapheme_root', 'vowel_diacritic']\n#     submission = DataFrame(columns=['row_id', 'target'])\n#     for index, row in df.iterrows():\n#         for output_field in output_fields:\n#             try:\n#                 index = f\"Test_{index}\" if not str(index).startswith('T') else index\n#                 submission = submission.append({\n#                     'row_id': f\"{index}_{output_field}\",\n#                     'target': df[output_field].loc[index],\n#                     }, ignore_index=True)\n#             except Exception as exception:\n#                 print('df_to_submission()', exception)\n#     print('df_to_submission() - output', submission.shape)\n#     return submission\n\n\n# def df_to_submission(df: DataFrame) -> DataFrame:\n#     print('df_to_submission_columns() - input', df.shape)\n#     output_fields = ['consonant_diacritic', 'grapheme_root', 'vowel_diacritic']\n#     submissions = {}\n#     for output_field in output_fields:\n#         if 'image_id' in df.columns:\n#             submissions[output_field] = DataFrame({\n#                 'row_id': df['image_id'] + '_' + output_field,\n#                 'target': df[output_field],\n#             })\n#         else:\n#             submissions[output_field] = DataFrame({\n#                 'row_id': df.index + '_' + output_field,\n#                 'target': df[output_field],\n#             })\n#\n#     # Kaggle - Order of submission.csv IDs matters - https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69366\n#     submission = DataFrame(pd.concat(submissions.values()))\n#     submission['sort'] = submission['row_id'].apply(lambda row_id: int(re.sub(r'\\D', '', row_id)) )\n#     submission = submission.sort_values(by=['sort','row_id'])\n#     submission = submission.drop(columns=['sort'])\n#\n#     print('df_to_submission_columns() - output', submission.shape)\n#     return submission\n\n\ndef df_to_submission(df: DataFrame) -> DataFrame:\n    print('df_to_submission_columns() - input', df.shape)\n    output_fields = ['consonant_diacritic', 'grapheme_root', 'vowel_diacritic']\n    if 'image_id' not in df.columns:\n        df['image_id'] = df.index\n\n    submission_rows = df.apply(lambda row: [{\n        'row_id': row['image_id'] + '_' + output_field,\n        'target': row[output_field],\n    } for output_field in output_fields], axis=1, result_type='reduce' )\n\n    submission = DataFrame(chain(*submission_rows.values))   # Hopefully in original sort order\n\n    print('df_to_submission_columns() - output', submission.shape)\n    return submission\n\n\n\ndef df_to_submission_csv(df: DataFrame, filename: str):\n    submission = df_to_submission(df)\n\n    os.makedirs(os.path.dirname(filename), exist_ok=True)\n    if os.environ.get('KAGGLE_KERNEL_RUN_TYPE'):\n        submission.to_csv('submission.csv', index=False)\n        print(\"wrote:\", 'submission.csv', submission.shape)\n    else:\n        submission.to_csv(filename, index=False)\n        print(\"wrote:\", filename, submission.shape)\n\n\n#####\n##### END   src/util/csv.py\n#####\n\n#####\n##### START src/util/hparam.py\n#####\n\nimport math\nimport os\nimport re\nimport time\nfrom typing import AnyStr, Dict, Union\n\nimport tensorflow as tf\nfrom tensorboard.plugins.hparams.api import KerasCallback\nfrom tensorflow.keras.backend import categorical_crossentropy\nfrom tensorflow.keras.callbacks import EarlyStopping, LearningRateScheduler, ModelCheckpoint, ReduceLROnPlateau\n\n# from src.callbacks.KaggleTimeoutCallback import KaggleTimeoutCallback\n# from src.dataset.DatasetDF import DatasetDF\n# from src.settings import settings\n# from src.util.logs import model_stats_from_history\n# from src.vendor.CLR.clr_callback import CyclicLR\n\n\n\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'  # 0, 1, 2, 3 # Disable Tensortflow Logging\n# tf.keras.backend.set_floatx('float16')  # Potentially causes problems with Tensortflow\n\n\ndef hparam_key(hparams):\n    return \"-\".join( f\"{key}={value}\" for key,value in sorted(hparams.items()) ).replace(' ','')\n\n\ndef min_lr(hparams):\n    # tensorboard --logdir logs/convergence_search/min_lr-optimized_scheduler-random-scheduler/ --reload_multifile=true\n    # There is a high degree of randomness in this parameter, so it is hard to distinguish from statistical noise\n    # Lower min_lr values for CycleCR tend to train slower\n    hparams = { **settings['hparam_defaults'], **hparams }\n    if 'min_lr'  in hparams:              return hparams['min_lr']\n    if hparams[\"optimizer\"] == \"SGD\":     return 1e-05  # preferred by SGD\n    else:                                 return 1e-03  # fastest, least overfitting and most accidental high-scores\n\n\n# DOCS: https://ruder.io/optimizing-gradient-descent/index.html\ndef scheduler(hparams: dict, dataset: DatasetDF, verbose=False):\n    hparams = { **settings['hparam_defaults'], **hparams }\n    if hparams['scheduler'] is 'constant':\n        return LearningRateScheduler(lambda epocs: hparams['learning_rate'], verbose=False)\n\n    if hparams['scheduler'] is 'linear_decay':\n        return LearningRateScheduler(\n            lambda epocs: max(\n                hparams['learning_rate'] * (10. / (10. + epocs)),\n                min_lr(hparams)\n            ),\n            verbose=verbose\n        )\n\n    if hparams['scheduler'].startswith('CyclicLR') \\\n            or hparams['scheduler'] in [\"triangular\", \"triangular2\", \"exp_range\"]:\n        # DOCS: https://www.datacamp.com/community/tutorials/cyclical-learning-neural-nets\n        # CyclicLR_triangular, CyclicLR_triangular2, CyclicLR_exp_range\n        mode = re.sub(r'^CyclicLR_', '', hparams['scheduler'])\n\n        # step_size should be epoc multiple between 2 and 8, but multiple of 2 (= full up/down cycle)\n        if   hparams['patience'] <=  6: whole_cycles = 1   #  1/2   = 0.5  | 6/2    = 3\n        elif hparams['patience'] <= 12: whole_cycles = 2   #  8/4   = 2    | 12/4   = 3\n        elif hparams['patience'] <= 24: whole_cycles = 3   # 14/6   = 2.3  | 24/6   = 4\n        elif hparams['patience'] <= 36: whole_cycles = 4   # 26/8   = 3.25 | 36/8   = 4.5\n        elif hparams['patience'] <= 48: whole_cycles = 5   # 28/10  = 2.8  | 48/10  = 4.8\n        elif hparams['patience'] <= 72: whole_cycles = 6   # 50/12  = 4.2  | 72/12  = 6\n        elif hparams['patience'] <= 96: whole_cycles = 8   # 74/16  = 4.6  | 96/16  = 6\n        else:                           whole_cycles = 12  # 100/24 = 4.2  | 192/24 = 8\n\n        return CyclicLR(\n            mode      = mode,\n            step_size =dataset.epoch_size() * (hparams['patience'] / (2.0 * whole_cycles)),\n            base_lr   = min_lr(hparams),\n            max_lr    = hparams['learning_rate']\n        )\n\n    if hparams['scheduler'].startswith('plateau'):\n        factor = int(( re.findall(r'\\d+', hparams['scheduler']) + [10] )[0])            # plateau2      || plateau10 (default)\n        if 'sqrt' in hparams['scheduler']:  patience = math.sqrt(hparams['patience'])  # plateau2_sqrt || plateau10__sqrt\n        else:                               patience = hparams['patience'] / 2.0\n\n        return ReduceLROnPlateau(\n            monitor  = 'val_loss',\n            factor   = 1 / factor,\n            patience = math.floor(patience),\n            min_lr   = 0,   # min_lr(train_hparams),\n            verbose  = verbose,\n        )\n\n    print(\"Unknown scheduler: \", hparams)\n\n\ndef losses(output_shape):\n    if   isinstance(output_shape, list): losses = [ categorical_crossentropy      for n   in output_shape        ]\n    elif isinstance(output_shape, dict): losses = { key: categorical_crossentropy for key in output_shape.keys() }\n    else:                                losses = categorical_crossentropy\n    return losses\n\n\ndef loss_weights(output_shape):\n    # unique = dataset.apply(lambda col: col.nunique()); unique\n    # grapheme_root           168   | sqrt = 12.9 / 54.9 = 0.24\n    # vowel_diacritic          11   | sqrt =  3.3 / 54.9 = 0.06\n    # consonant_diacritic       7   | sqrt =  2.6 / 54.9 = 0.05\n    # grapheme               1295   | sqrt = 35.9 / 54.9 = 0.65\n    if not isinstance(output_shape, dict): return None\n    norm    = sum(map(math.sqrt, output_shape.values()))\n    weights = {\n        key: math.sqrt(value)/norm\n        for key,value in output_shape.items()\n    }\n    return weights\n\n\n\ndef callbacks(hparams, dataset, model_file=None, log_dir=None, best_only=True, verbose=False ):\n    schedule  = scheduler(hparams, dataset, verbose=verbose)\n\n    callbacks = [\n        EarlyStopping(\n            monitor='val_loss',\n            mode='min',\n            verbose=verbose,\n            patience=hparams.get('patience', 10),\n            restore_best_weights=best_only\n        ),\n        schedule,\n    ]\n    if hparams.get(\"timeout\"):\n        callbacks += [\n            KaggleTimeoutCallback( hparams.get(\"timeout\"), verbose=False ),\n        ]\n    if model_file:\n        callbacks += [\n            ModelCheckpoint(\n                model_file,\n                monitor='val_loss',\n                verbose=False,\n                save_best_only=best_only,\n                save_weights_only=False,\n                mode='auto',\n            )\n        ]\n    if log_dir and settings['verbose']['tensorboard'] and not os.environ.get('KAGGLE_KERNEL_RUN_TYPE'):\n        callbacks += [\n            tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1),  # log metrics\n            KerasCallback(log_dir, hparams)                                     # log train_hparams\n        ]\n    return callbacks\n\n\n\ndef model_compile(\n        hparams:      Dict,\n        model:        tf.keras.models.Model,\n        output_shape: Union[None, int, Dict] = None,\n    ):\n    hparams   = { **settings['hparam_defaults'], **hparams }\n    optimiser = getattr(tf.keras.optimizers, hparams['optimizer'])\n    loss      = losses(output_shape)\n    weights   = loss_weights(output_shape) if hparams.get('loss_weights') else None\n\n    model.compile(\n        loss=loss,\n        loss_weights=weights,\n        optimizer=optimiser(learning_rate=hparams.get('learning_rate', 0.001)),\n        metrics=['accuracy']\n    )\n    return model\n\n\ndef model_compile_fit(\n        hparams:      Dict,\n        model:        tf.keras.models.Model,\n        dataset:      DatasetDF,\n        epochs      = 999,\n        output_shape: Union[None, int, Dict] = None,\n        model_file:   AnyStr = None,\n        log_dir:      AnyStr = None,\n        best_only   = True,\n        verbose     = settings['verbose']['fit'],\n):\n    timer_start = time.time()\n\n    hparams = { **settings['hparam_defaults'], **hparams }\n    model   = model_compile( hparams, model, output_shape )\n\n    callback = callbacks(hparams, dataset, model_file, log_dir, best_only, verbose)\n    history  = model.fit(\n        dataset.X[\"train\"], dataset.Y[\"train\"],\n        batch_size=hparams.get(\"batch_size\", 128),\n        epochs=epochs,\n        verbose=verbose,\n        validation_data=(dataset.X[\"valid\"], dataset.Y[\"valid\"]),\n        callbacks=callback\n    )\n    timer_seconds = int(time.time() - timer_start)\n\n    model_stats = model_stats_from_history(history, timer_seconds, best_only)\n    return model_stats\n\n\n#####\n##### END   src/util/hparam.py\n#####\n\n#####\n##### START src/pipelines/image_data_generator_cnn.py\n#####\n\n#!/usr/bin/env python\nimport os\nimport time\n\nimport glob2\nimport numpy as np\nfrom pyarrow.parquet import ParquetFile\n\n# from src.dataset.DatasetDF import DatasetDF\n# from src.dataset.ParquetImageDataGenerator import ParquetImageDataGenerator\n# from src.dataset.Transforms import Transforms\n# from src.models.MultiOutputCNN import MultiOutputCNN\n# from src.settings import settings\n# from src.util.argparse import argparse_from_dicts\n# from src.util.csv import df_to_submission_csv, submission_df_generator\n# from src.util.hparam import callbacks, hparam_key, model_compile, model_stats_from_history\n# from src.util.logs import log_model_stats\n\n\n\ndef image_data_generator_cnn(\n        train_hparams,\n        model_hparams,\n        pipeline_name,\n        model_file=None,\n        log_dir=None,\n        verbose=2,\n        load_weights=True\n):\n    combined_hparams = { **model_hparams, **train_hparams }\n    train_hparams    = { **settings['hparam_defaults'], **train_hparams }\n    print(\"pipeline_name\", pipeline_name)\n    print(\"train_hparams\", train_hparams)\n    print(\"model_hparams\", model_hparams)\n\n    model_hparams_key = hparam_key(model_hparams)\n    train_hparams_key = hparam_key(train_hparams)\n\n    # csv_data    = pd.read_csv(f\"{settings['dir']['data']}/train.csv\")\n    model_file  = model_file or f\"{settings['dir']['models']}/{pipeline_name}/{pipeline_name}-{model_hparams_key}.hdf5\"\n    log_dir     = log_dir    or f\"{settings['dir']['logs']}/{pipeline_name}/{model_hparams_key}/{train_hparams_key}\"\n\n    os.makedirs(os.path.dirname(model_file), exist_ok=True)\n    os.makedirs(log_dir,                     exist_ok=True)\n\n    dataset_rows = ParquetFile(f\"{settings['dir']['data']}/train_image_data_0.parquet\").metadata.num_rows\n    dataset      = DatasetDF(size=1)\n    input_shape  = dataset.input_shape()\n    output_shape = dataset.output_shape()\n    model = MultiOutputCNN(\n        input_shape=input_shape,\n        output_shape=output_shape,\n        **model_hparams,\n    )\n    model_compile(model_hparams, model, output_shape)\n\n    # Load Pre-existing weights\n    if load_weights:\n        if os.path.exists( model_file ):\n            try:\n                model.load_weights( model_file )\n                print('Loaded Weights: ', model_file)\n            except Exception as exception: print('exception', exception)\n\n        if os.environ.get('KAGGLE_KERNEL_RUN_TYPE'):\n            load_models = (glob2.glob(f'../input/**/{os.path.basename(model_file)}')\n                        +  glob2.glob(f'../input/**/{os.path.basename(model_file)}'.replace('=','')))  # Kaggle Dataset Upload removes '='\n            for load_model in load_models:\n                try:\n                    model.load_weights( load_model )\n                    print('Loaded Weights: ', load_model)\n                    # break\n                except Exception as exception: print('exception', exception)\n\n    if verbose:\n        model.summary()\n\n    # Source: https://www.kaggle.com/jamesmcguigan/bengali-ai-image-processing\n    datagen_args = {\n        # \"rescale\":          1./255,  # \"normalize\": True is default in Transforms\n        \"zoom_range\":         0.2,\n        \"width_shift_range\":  0.1,     # we already have centering\n        \"height_shift_range\": 0.1,     # we already have centering\n        \"rotation_range\":     45/2,\n        \"shear_range\":        45/2,\n        # \"brightness_range\":   0.5,   # Prebrightness normalized\n        \"fill_mode\":         'constant',\n        \"cval\": 0,\n        # \"featurewise_center\": True,             # No visible effect in plt.imgshow()\n        # \"samplewise_center\": True,              # No visible effect in plt.imgshow()\n        # \"featurewise_std_normalization\": True,  # No visible effect in plt.imgshow() | requires .fit()\n        # \"samplewise_std_normalization\": True,   # No visible effect in plt.imgshow() | requires .fit()\n        # \"zca_whitening\": True,                   # Kaggle, insufficent memory\n    }\n    flow_args = {}\n    flow_args['train'] = {\n        \"transform_X\":      Transforms.transform_X,\n        \"transform_X_args\": {},  #  \"normalize\": True is default in Transforms\n        \"transform_Y\":      Transforms.transform_Y,\n        \"batch_size\":       train_hparams['batch_size'],\n        \"reads_per_file\":   2,\n        \"resamples\":        1,\n        \"shuffle\":          True,\n        \"infinite\":         True,\n    }\n    flow_args['valid'] = {\n        **flow_args['train'],\n        \"resamples\":  1,\n    }\n    flow_args['test'] = {\n        **flow_args['train'],\n        \"resamples\":  1,\n        \"shuffle\":    False,\n        \"infinite\":   False,\n        \"test\":       True,\n    }\n\n    datagens = {\n        \"train\": ParquetImageDataGenerator(**datagen_args),\n        \"valid\": ParquetImageDataGenerator(),\n        \"test\":  ParquetImageDataGenerator(),\n    }\n    # [ datagens[key].fit(train_batch) for key in datagens.keys() ]  # Not required\n    fileglobs = {\n        \"train\": f\"{settings['dir']['data']}/train_image_data_[123].parquet\",\n        \"valid\": f\"{settings['dir']['data']}/train_image_data_0.parquet\",\n        \"test\":  f\"{settings['dir']['data']}/test_image_data_*.parquet\",\n    }\n    ### Preserve test/train split for Kaggle\n    # if os.environ.get('KAGGLE_KERNEL_RUN_TYPE'):\n    #     # For the Kaggle Submission, train on all available data and rely on Kaggle Timeout\n    #     fileglobs[\"train\"] = f\"{settings['dir']['data']}/train_image_data_*.parquet\"\n\n    generators = {\n        key: datagens[key].flow_from_parquet(value, **flow_args[key])\n        for key,value in fileglobs.items()\n    }\n    dataset_rows_per_file = {\n        key: np.mean([ ParquetFile(filename).metadata.num_rows for filename in glob2.glob(fileglobs[key]) ])\n        for key in fileglobs.keys()\n    }\n    dataset_rows_total = {\n        key: sum([ ParquetFile(filename).metadata.num_rows for filename in glob2.glob(fileglobs[key]) ])\n        for key in fileglobs.keys()\n    }\n\n    ### Epoch: train == one whole parquet files | valid = 1 filesystem read\n    steps_per_epoch  = int(dataset_rows_per_file['train'] / flow_args['train']['batch_size'] * flow_args['train']['resamples'] )\n    validation_steps = int(dataset_rows_per_file['valid'] / flow_args['valid']['batch_size'] / flow_args['train']['reads_per_file'] )\n    callback         = callbacks(combined_hparams, dataset, model_file, log_dir, best_only=True, verbose=1)\n\n    timer_start = time.time()\n    history = model.fit(\n        generators['train'],\n        validation_data  = generators['valid'],\n        epochs           = train_hparams['epochs'],\n        steps_per_epoch  = steps_per_epoch,\n        validation_steps = validation_steps,\n        verbose          = verbose,\n        callbacks        = callback\n    )\n    timer_seconds = int(time.time() - timer_start)\n    model_stats   = model_stats_from_history(history, timer_seconds, best_only=True)\n\n    return model, model_stats, output_shape\n\n\n\n\nif __name__ == '__main__':\n    # Fastest with high score\n    # - maxpool_layers=5 | cnns_per_maxpool=3 | dense_layers=1 | dense_units=256 | global_maxpool=False | regularization=False\n    #\n    # Shortlist:\n    # - maxpool_layers=5 | cnns_per_maxpool=3 | dense_layers=1 | dense_units=512 | global_maxpool=True  | regularization=False\n    # - maxpool_layers=4 | cnns_per_maxpool=4 | dense_layers=1 | dense_units=256 | global_maxpool=False | regularization=False\n    # - maxpool_layers=4 | cnns_per_maxpool=4 | dense_layers=1 | dense_units=256 | global_maxpool=False | regularization=True\n    model_hparams = {\n        \"cnns_per_maxpool\":   4,\n        \"maxpool_layers\":     4,\n        \"dense_layers\":       1,\n        \"dense_units\":      256,\n        \"regularization\":  True,\n        \"global_maxpool\": False,\n    }\n    train_hparams = {\n        \"optimizer\":     \"Adadelta\",\n        \"scheduler\":     \"plateau10\",\n        \"learning_rate\": 1,\n        \"patience\":      20,\n        \"best_only\":     True,\n        \"batch_size\":    128,     # Too small and the GPU is waiting on the CPU - too big and GPU runs out of RAM - keep it small for kaggle\n        \"epochs\":        999,\n        \"loss_weights\":  False,\n    }\n    if os.environ.get('KAGGLE_KERNEL_RUN_TYPE') == 'Interactive':\n        train_hparams['patience'] = 0\n        train_hparams['epochs']   = 1\n    train_hparams = { **settings['hparam_defaults'], **train_hparams }\n\n    argparse_from_dicts([train_hparams, model_hparams], inplace=True)\n\n\n    pipeline_name     = \"image_data_generator_cnn\"\n    model_hparams_key = hparam_key(model_hparams)\n    train_hparams_key = hparam_key(train_hparams)\n    logfilename       = f\"{settings['dir']['submissions']}/{pipeline_name}-{model_hparams_key}-submission.log\"\n    csv_filename      = f\"{settings['dir']['submissions']}/{pipeline_name}-{model_hparams_key}-submission.csv\"\n\n    model, model_stats, output_shape = image_data_generator_cnn(train_hparams, model_hparams, pipeline_name, load_weights=True)\n\n    log_model_stats(model_stats, logfilename, model_hparams, train_hparams)\n\n    submission = submission_df_generator(model, output_shape)\n    df_to_submission_csv( submission, csv_filename )\n\n\n#####\n##### END   src/pipelines/image_data_generator_cnn.py\n#####\n\n##### \n##### ./kaggle_compile.py src/pipelines/image_data_generator_cnn.py --commit\n##### \n##### 2020-03-31 18:06:42+01:00\n##### \n##### origin\tgit@github.com:JamesMcGuigan/kaggle-bengali-ai.git (fetch)\n##### origin\tgit@github.com:JamesMcGuigan/kaggle-bengali-ai.git (push)\n##### \n##### * master b647190 [ahead 1] image_data_generator_cnn | Preserve test/train split for Kaggle\n##### \n##### b647190f8cbd70b31f472ea258be821ec0b3401d\n##### + 4/4/1 + Dont train on grapheme\n##### + Regularization","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}