{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nAs a beginner, I made this notebook to present a generic approach to \"play\" with the concepts of machine learning and neural network. I have also tried to provide some clean Python code. To sum up, it is a synthesis of my current knowledge (and sorry for my English).\n\nThis is a first version which will be improved compete after compete.\n\nThe basic steps to define a 'Generic approach of Machine Learning' are:\n1. Define the problem, I mean understand the data you got and define what are the inputs (attributes) and what is the output (target) of your Machine Learning\n2. Summarize the dataset content in a statistical form\n3. Prepare the dataset for your Machine Learning processing\n4. Evaluate a set of algorithms based on you understanding of the data\n5. Improve the results of your Machine Learning by refining the algorithms\n6. Present the results of your Machine Learning\n7. Deploy or save your Machine Learning\n\n**NOTE: Please, feel free to correct and enhance this notebook ;)**","metadata":{}},{"cell_type":"markdown","source":"Before to start our notebook, we have to upgrade Python installer and to install some additional modules.","metadata":{}},{"cell_type":"markdown","source":"To define the problem, we have first to choose the subject we will work on. The point 1.b provides different datasets we can use to play. For each dataset, a comment describes the problem to address. \nWe will consider two different problems:\n1. One about classification (the basic one is the Iris classification)\n2. One about regression (Melbourne housing prices)\n\nThis is the part that cannot be generic. The generic behavior proposed here is parameterized by the set of parameters defined in point b.1.\n\nNote: In point b.1, to swith to another problem, just comment the current one and uncomment the problem to play with\n\nSwitching to Python code, the first step is to load all the required libraries (1.a) and to choose the problem to solve, let's say Iris classification or Melbourne House Prices regression.","metadata":{}},{"cell_type":"code","source":"from __future__ import division # Import floating-point division (1/4=0.25) instead of Euclidian division (1/4=0)\n\n# 1. Prepare Problem\n\n# a) Load libraries\nimport os, warnings, argparse, io, operator, requests, math, random, tempfile\nimport re # Regular expressions support\nfrom datetime import datetime # Date & Time support\n\nimport numpy as np # Linear algebra\nimport matplotlib.pyplot as plt # Data visualization\nimport seaborn as sns # Enhanced data visualization\n#import seaborn_image as isns # Enhanced image data visualization\nimport pandas as pd # Data processing, CSV file I/O (e.g. pd.read_csv)\n\nfrom pandas_profiling import ProfileReport\n\nimport sklearn\nfrom sklearn import model_selection\nfrom sklearn import linear_model # Regression\nfrom sklearn import discriminant_analysis\nfrom sklearn import neighbors # Clustering\nfrom sklearn import naive_bayes\nfrom sklearn import tree # Decisional tree learning\nfrom sklearn import svm # Support Vector Machines\nfrom sklearn import ensemble # Support RandomForestRegressor, GradientBoostingRegressor, ExtraTreesRegressor, AdaBoostRegressor\n\nimport xgboost as xgb # Gradient Boosted Decision Trees algorithm\n\nimport lightgbm as lgb # Light Gradient Boost model\n\nfrom sklearn.base import is_classifier, is_regressor # To check if the model is for regression or classification (do not work for Keras)\n\nfrom sklearn.impute import SimpleImputer \n\nfrom sklearn.preprocessing import LabelEncoder # Labelling categorical feature from 0 to N_class - 1('object' type)\nfrom sklearn.preprocessing import LabelBinarizer # Binary labelling of categorical feature\nfrom sklearn.preprocessing import OneHotEncoder\nfrom sklearn.preprocessing import MultiLabelBinarizer\n\nfrom sklearn.preprocessing import StandardScaler # Data normalization\nfrom sklearn.preprocessing import MinMaxScaler # Data normalization\nfrom sklearn.preprocessing import MaxAbsScaler # Data normalization\n\nfrom sklearn.pipeline import Pipeline\n\nfrom sklearn.metrics import classification_report\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.metrics import r2_score\nfrom sklearn.metrics import f1_score\nfrom sklearn.metrics import precision_score\nfrom sklearn.metrics import recall_score\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.metrics import mean_squared_error\n\nfrom sklearn.model_selection import GridSearchCV\n\nfrom sklearn.feature_selection import mutual_info_regression, mutual_info_classif # To build Mutual Information plots\n\nfrom sklearn.inspection import permutation_importance\n\nimport pickle # Use to save and load models\n\nimport eli5\nfrom eli5.sklearn import PermutationImportance\n\n# Neural Network\nimport tensorflow as tf\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau\nimport keras\nfrom keras.wrappers.scikit_learn import KerasClassifier, KerasRegressor\nfrom keras.preprocessing.image import ImageDataGenerator, load_img, img_to_array, array_to_img\n\nfrom PIL import Image","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First of all, we have to define the problem:\n1. Understand the data, see point b.1) below\n2. Prepare the basics of your code such as loading the libraries and your data, see points a) and b) below\n\nIn point b.1, we have a set of parameters strongly linked to the problem to solve. These parameters are used to configure the execution of 'Generic approach of Machine Learning':\n- ML_NAME: The name of the Machine Learning (e.g. Titanic, Pima or Iris)\n- DATABASE_URI: The root directory of the datasets (e.g. tpu-getting-started for Flower Classification with TPUs compete)\n- DATABASE_NAME: The base name of the Images database\n- COLUMNS_LABEL: Columns label of the dataset. Default: None, means that labels are already present in the loaded dataset\n- TARGET_COLUMNS: The output column\n- OUTPUT_IS_REGRESSION: Indicates if the ML is about either regression (True) or classification (False)\n- DATE_TIME_COLUMNS: The list of the date/time column in customized format such as string format\n- EXCLUDE_FROM_OULIERS: features to exclude from Ouliers processing\n- NON_TRANSFORMABLE_COLUMNS: Indicates a list of columns which shall not be included in the transformation process (point 3.b)","metadata":{}},{"cell_type":"markdown","source":"# Description of the different projects of my playground","metadata":{}},{"cell_type":"code","source":"# b) Helpers\n\n# Set execution control flags\nfrom enum import IntFlag\nclass ExecutionFlags(IntFlag):\n    \"\"\"\n    This class provides some execution control flags to enable/disable some part of the whole script execution\n    \"\"\"\n    NONE                         = 0b00000000 # All flags disabled\n    ALL                          = 0b11111111 # All flags enabled\n    DATA_STAT_SUMMURIZE_FLAG     = 0b00000001 # Enable statitistical analyzis\n    DATA_VISUALIZATION_FLAG      = 0b00000010 # Enable data visualization\n    DATA_CLEANING_FLAG           = 0b00000100 # Enable data cleaning (feature engineering)\n    DATA_TRANSFORM_FLAG          = 0b00001000 # Enable data transformation\n    USE_NEURAL_NETWORK_FLAG      = 0b00010000 # Enable neural network models for Machine Learning\n    USE_ONLY_NEURAL_NETWORK_FLAG = 0b00100000 # Use only neural network models for Machine Learning\n    USE_CNN_NEURAL_NETWORK_FLAG  = 0b01000000 # Use neural network models for images based learning\n                                              # This flag exclude all the others\n    # End of class ExecutionFlags\n    \n# b.1) Define global parameters\n# Regression\n\n# Jan Tabular Playground Competition\n\"\"\"\nML_NAME = 'JanTabularPlaygroundCompetition'\nDATABASE_URI = None\nCOLUMNS_LABEL = None\nCOLUMNS_TO_DROP = ['id'] # Id is useless\nTARGET_COLUMNS = 'target'\nOUTPUT_IS_REGRESSION = True\nDATE_TIME_COLUMNS = None\nNON_TRANSFORMABLE_COLUMNS = None\n\"\"\"\n# To predict house price using the famous Melbourne housing dataset\n\"\"\"\nML_NAME = 'MelbourneHousing'\nDATABASE_URI = 'https://raw.githubusercontent.com/nagoya-foundation/r-functions-performance/master/data/Melbourne_housing_FULL.csv'\nCOLUMNS_LABEL = None\nCOLUMNS_TO_DROP = ['Address', 'Method', 'Postcode', 'CouncilArea', 'Propertycount', 'Regionname', 'SellerG', 'Suburb']\nTARGET_COLUMNS = 'Price'\nOUTPUT_IS_REGRESSION = True\nDATE_TIME_COLUMNS = ['Date']\nEXCLUDE_FROM_OULIERS = ['Lattitude', 'Longtitude']\nNON_TRANSFORMABLE_COLUMNS = ['Lattitude', 'Longtitude']\n# Suburb\n# Address\n# Rooms\n# Type\n# Price\n# Method\n# SellerG\n# Date\n# Distance\n# Postcode\n# Bedroom2\n# Bathroom\n# Car\n# Landsize\n# BuildingArea\n# YearBuilt\n# CouncilArea\n# Lattitude\n# Longtitude\n# Regionname\n# Propertycount\n\"\"\"\n\n# Classification\n\n# To categorize an iris flower according to the dimensions of its sepals & petals \n\"\"\"\n# Famous database; from Fisher, 1936\nML_NAME = 'Iris'\nDATABASE_URI = 'https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'\nCOLUMNS_LABEL = ['sepal length in cm', 'sepal width in cm', 'petal length in cm', 'petal width in cm', 'class']\nCOLUMNS_TO_DROP = None\nTARGET_COLUMNS = 'class'\nOUTPUT_IS_REGRESSION = False\nDATE_TIME_COLUMNS = None\nEXCLUDE_FROM_OULIERS = None\nNON_TRANSFORMABLE_COLUMNS = None\n\"\"\"\n\n# To predict survival on the Titanic\n\"\"\"\nML_NAME = 'Titanic' # https://www.kaggle.com/c/titanic\nDATABASE_URI = 'https://raw.githubusercontent.com/alexisperrier/packt-aml/master/ch4/titanic.csv'\nTEST_FILE_NAME = '../input/titanic/test.csv'\nCOLUMNS_LABEL = None\nCOLUMNS_TO_DROP = ['PassengerId', 'Name', 'Ticket'] # PassengerId is useless, Name and Ticket will be processed in future version\n    # We assume that Name,Ticket and are not relevant information\n    # This can be confirm by the correlation matrix\nTARGET_COLUMNS = 'Survived'\nOUTPUT_IS_REGRESSION = False\nDATE_TIME_COLUMNS = None\nEXCLUDE_FROM_OULIERS = None\nNON_TRANSFORMABLE_COLUMNS = None\n#  PassengerId: Unique passenger id\n#  Survived: Survival status ('Yes' or 'No')\n#  Pclass: The class the passeger belong (1st, 2nd or 3rd class)\n#  Name: Name of the passenger\n#  Sex: The sex of the passenger ('male' of 'female')\n#  Age: The age of the passenger (in years)\n#  SibSp: # of siblings / spouses aboard the Titanic\n#  Parch: # of parents / children aboard the Titanic\n#  Ticket: No description available for this field, perhaps the travel company identifier\n#  Fare: Ticket price\n#  Cabin: Identifier of the cabin. The first character identifies the deck.\n#         This could be interesting fo the ML, creating a new feature Deck\n#  Embarked: Port of Embarkation\n\"\"\"\n\n# This dataset describes the medical records for Pima Indians and whether or not each patient will have an onset of diabetes within \fve years.\n\"\"\"\n# NOTE: Disable flag DATA_CLEANING_FLAG, this dataset is already ready to be used by ML \nML_NAME = 'Pima'\nDATABASE_URI = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv'\nTEST_FILE_NAME = None\nCOLUMNS_LABEL = ['preg', 'plas', 'pres (mm Hg)', 'skin (mm)', 'test (mu U/ml)', 'mass', 'pedi', 'age (years)', 'class']\nTARGET_COLUMNS = 'class'\nOUTPUT_IS_REGRESSION = False\nDATE_TIME_COLUMNS = None\nDATE_TIME_COLUMNS = None\nEXCLUDE_FROM_OULIERS = None\nNON_TRANSFORMABLE_COLUMNS = None\n# Use Neural Network for Machine Learning\n#FLAGS = ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG & ExecutionFlags.DATA_TRANSFORM_FLAG & ExecutionFlags.DATA_VISUALIZATION_FLAG\n# Use standard Machine Learning\nFLAGS = ExecutionFlags.ALL & ~ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG & ~ExecutionFlags.USE_NEURAL_NETWORK_FLAG & ~ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG \\\n        #& ~ExecutionFlags.DATA_TRANSFORM_FLAG \\\n        #& ~ExecutionFlags.DATA_STAT_SUMMURIZE_FLAG \\\n        #& ~ExecutionFlags.DATA_VISUALIZATION_FLAG \\\n        # Keep it empty for\n#  preg = Number of times pregnant\n#  plas = Plasma glucose concentration a 2 hours in an oral glucose tolerance test\n#  pres = Diastolic blood pressure\n#  skin = Triceps skin fold thickness (mm)\n#  test = 2-Hour serum insulin (mu U/ml)\n#  mass = Body mass index (weight in kg/(height in m)^2)\n#  pedi = Diabetes pedigree function\n#  age = Age (years)\n#  class = Class variable (1:tested positive for diabetes, 0: tested negative for diabetes)\n\"\"\"\n\n# This dataset collects information from 100k medical appointments in Brazil and is focused on the question of whether or not patients show up for their appointment. A number of characteristics about the patient are included in each row\n\"\"\"\nML_NAME = 'BrazilMedicalAppointments'\nDATABASE_URI = 'https://github.com/jbrownlee/Datasets/blob/master/pima-indians-diabetes.data.csv'\nTARGET_COLUMNS = 'No-show'\nCOLUMNS_LABEL = None\nOUTPUT_IS_REGRESSION = False\nCOLUMNS_TO_DROP = None\nDATE_TIME_COLUMNS = None\nEXCLUDE_FROM_OULIERS = None\nNON_TRANSFORMABLE_COLUMNS = None\n\"\"\"\n\n# This dataset describes flower classification, using TPU.\n\"\"\"\n# Require ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG\nML_NAME = 'FlowerClassification' # https://www.kaggle.com/c/tpu-getting-started\nDATABASE_NAME = 'tpu-getting-started'\nIMAGES_SAMPLES_NUM = 512 # Total number of images to be used for all datasets. None: Use all images\nIMAGE_NUM_PIXELS = 512 # Image size in pixels\nIMAGE_SIZE = [IMAGE_NUM_PIXELS, IMAGE_NUM_PIXELS] # (Heigh,Width) image size in pixels\nIMAGE_CLASSES = [ # Classes name\n                    'pink primrose', 'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n                    'snapdragon', 'colt''s foot',               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n                    'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n                    'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n                    'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n                    'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n                    'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n                    'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n                    'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n                    'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n                    'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose'\n                ]\nTARGET_COLUMNS = None\nOUTPUT_IS_REGRESSION = False\nDATE_TIME_COLUMNS = None\nDATE_TIME_COLUMNS = None\nEXCLUDE_FROM_OULIERS = None\nNON_TRANSFORMABLE_COLUMNS = None\nFLAGS = ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG # | ExecutionFlags.DATA_VISUALIZATION_FLAG # Use DL woth CNN (Images) \n\"\"\"\n\n# Bristol-Myers Squibb – Molecular Translation\n\"\"\"\n# Require ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG\nML_NAME = 'BMS-MolecularTranslation' # https://www.kaggle.com/c/bms-molecular-translation\nDATABASE_NAME = 'bms-molecular-translation'\nCOLUMNS_LABEL = ['image_id', 'class']\nIMAGES_SAMPLES_NUM = None # Total number of images to be used for all datasets. None: Use all images\nIMAGE_NUM_PIXELS = 512 # Image size in pixels\nIMAGE_SIZE = [IMAGE_NUM_PIXELS, IMAGE_NUM_PIXELS] # (Heigh,Width) image size in pixels\nIMAGE_CLASSES = None\nTARGET_COLUMNS = None\nOUTPUT_IS_REGRESSION = False\nDATE_TIME_COLUMNS = None\nDATE_TIME_COLUMNS = None\nEXCLUDE_FROM_OULIERS = None\nNON_TRANSFORMABLE_COLUMNS = None\nFLAGS = ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG || ExecutionFlags.DATA_VISUALIZATION_FLAG # Use DL woth CNN (Images) \n\"\"\"\n\n# Human Protein Atlas - Single Cell Classification (https://www.kaggle.com/c/hpa-single-cell-image-classification)\n# Require ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG\nML_NAME = 'HumanProteinAtlas' # https://www.kaggle.com/c/hpa-single-cell-image-classification\nDATABASE_NAME = 'hpa-single-cell-image-classification'\nCOLUMNS_LABEL = ['image_id', 'class']\nIMAGES_SAMPLES_NUM = 4096 # Total number of images to be used for all datasets. None: Use all images\nIMAGE_NUM_PIXELS = 512 # Image size in pixels\nIMAGE_SIZE = [IMAGE_NUM_PIXELS, IMAGE_NUM_PIXELS] # (Heigh,Width) image size in pixels\nIMAGE_CLASSES = [ # Classes name\n                    'Nucleoplasm', \n                    'Nuclear membrane', \n                    'Nucleoli', \n                    'Nucleoli fibrillar center', \n                    'Nuclear speckles', \n                    'Nuclear bodies', \n                    'Endoplasmic reticulum', \n                    'Golgi apparatus', \n                    'Intermediate filaments', \n                    'Actin filaments', \n                    'Microtubules', \n                    'Mitotic spindle', \n                    'Centrosome', \n                    'Plasma membrane', \n                    'Mitochondria', \n                    'Aggresome', \n                    'Cytosol', \n                    'Vesicles and punctate cytosolic patterns', \n                    'Negative'\n                ]\nTARGET_COLUMNS = None\nOUTPUT_IS_REGRESSION = False\nDATE_TIME_COLUMNS = None\nDATE_TIME_COLUMNS = None\nEXCLUDE_FROM_OULIERS = None\nNON_TRANSFORMABLE_COLUMNS = None\nFLAGS = ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG # | ExecutionFlags.DATA_VISUALIZATION_FLAG # Use DL woth CNN (Images)\n# Channels:\n# RED: Microtubule channels\n# GREEN: Protein of interest\n# BLUE: Nuclei channels\n# YELLOW: Endoplasmic reticulum\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Pre and Post processings\n\nThe functions below are some pre and post actions that will be executed during the different steps of the Machine Leaning/Deep Learning training and validation.\n\nThese functions have to be completed by yourself after learning from dataset Descriptive statistics (point 2.a) and dataset visualization (point 2.b) steps in order to:\n- Remove useless features after loading dataset (kaggle_post_load_datasets (point 1.c))\n- Apply some feature processing rugth after the datasets are loaded (kaggle_post_load_datasets (point 1.c))\n- Apply early features selection (kaggle_post_load_datasets (point 1.c))\n- Apply feature processing resulting of the data analyzing (kaggle_pre_features_engineering)\n- Create new features","metadata":{}},{"cell_type":"code","source":"def kaggle_pre_main() -> None:\n    print('----------------------------- kaggle_pre_main -----------------------------')\n    \"\"\"\n    This function is called at the begining of the main procedure \n    E.g. Install some specific packages...\n    \"\"\"\n    print('kaggle_pre_main: Done')\n\ndef kaggle_post_main() -> None:\n    \"\"\"\n    This function is called at the termination of the main procedure \n    E.g. Uninstall some specific packages, cleanup, publishing...\n    \"\"\"\n    print('----------------------------- kaggle_post_main -----------------------------')\n    print('kaggle_post_main: Done')\n\ndef kaggle_pre_load_datasets() -> None:\n    \"\"\"\n    This function is called by kaggle_load_datasets() just before to start processing\n    E.g. Rename or reorganize the datasets...\n    \"\"\"\n    print('----------------------------- kaggle_pre_load_datasets -----------------------------')\n    print('kaggle_pre_load_datasets: Done')\n\ndef kaggle_post_load_datasets(p_train_df, p_validation_df, p_test_df) -> list:\n    \"\"\"\n    This function is called by kaggle_load_datasets() just after the datasets were loaded\n    \"\"\"\n    print('----------------------------- kaggle_post_load_datasets -----------------------------')\n\n    # Refactor and enhance dataset content\n    if ML_NAME == 'Titanic':\n        print('kaggle_post_load_datasets: Replacing impossible 0 values by NaN value')\n        # Create a category U for Unknown and just keep the deck indetifier\n        p_train_df['Cabin'] = p_train_df['Cabin'].apply(lambda p_value : p_value[0:1] if not p_value is np.NaN else 'U') \n        p_validation_df['Cabin'] = p_validation_df['Cabin'].apply(lambda p_value : p_value[0:1] if not p_value is np.NaN else 'U') \n        p_test_df['Cabin'] = p_test_df['Cabin'].apply(lambda p_value : p_value[0:1] if not p_value is np.NaN else 'U') \n\n    # Drop columns if any\n    if not COLUMNS_TO_DROP is None:\n        p_train_df.drop(COLUMNS_TO_DROP, inplace = True, axis = 1)\n        p_validation_df.drop(COLUMNS_TO_DROP, inplace = True, axis = 1)\n        p_test_df.drop(COLUMNS_TO_DROP, inplace = True, axis = 1)\n    \n    # Keep only features selection\n    if ML_NAME == 'MelbourneHousing':\n        l = list(set(p_train_df.columns) - set(['Rooms', 'Bathroom', 'Landsize', 'BuildingArea', 'YearBuilt', 'Lattitude', 'Longtitude', 'Date', 'Distance', 'Car']))\n        l.remove(TARGET_COLUMNS)\n        if len(l) != 0:\n            p_train_df.drop(l, inplace = True, axis = 1) # train_df and validation_df contain TARGET column\n            p_validation_df.drop(l, inplace = True, axis = 1)\n            p_test_df.drop(l, inplace = True, axis = 1) # test_df does not\n        \n    print('kaggle_post_load_datasets: Done')\n    return p_train_df, p_validation_df, p_test_df\n\ndef kaggle_pre_load_images_datasets(p_train_url: str, \n                                    p_labels: list = None,\n                                    p_global_path: str = None,\n                                    p_validation_url: str = None,\n                                    p_test_url: str = None,\n                                    p_train_size: float = 0.9\n                                    ) -> list:\n    \"\"\"\n    This function is called by kaggle_load_images_datasets() just before to start processing\n    The Images database shall be organized as follow:\n    1) In Tfrecord format\n        <current_folder>/tfrecords-jpeg-<n>x<n>/train\n        <current_folder>/tfrecords-jpeg-<n>x<n>/val\n        <current_folder>/tfrecords-jpeg-<n>x<n>/test\n    2) Otherwise:\n        <current_folder>/train_labels.csv\n        <current_folder>/train\n        [<current_folder>/train], could not exist\n        <current_folder>/test\n    :parameters p_labels: The label of the columns to be used. Default: None\n    :parameter p_train_size: Size ratio of Training dataset vs. Validation dataset\n    \"\"\"\n    print('----------------------------- kaggle_pre_load_images_datasets -----------------------------')\n    train_df = None\n    validation_df = None\n    test_df = None\n    \n    if not p_train_url is None: # Case of separated Images and labels\n        print('kaggle_pre_load_images_datasets: Processing %s' % ML_NAME)\n        path = p_global_path #os.path.join(p_global_path, p_train_url) # p_train_url = DATABASE_NAME\n        if ML_NAME == 'HumanProteinAtlas':\n            def _convert_multi_labels(p_labels: list, p_length: int, p_splitter: str = '|') -> list:\n                labels = []\n                for label in p_labels:\n                    l = np.zeros(p_length).astype(dtype = np.uint8)\n                    for item in label:\n                        if(item != p_splitter):\n                            l[int(item)] = int(1)\n                    labels.append(list(l))\n                return labels\n                # End of function _convert_multi_labels\n\n            # Add a 'full path' feature with 4 color channels full path\n            p = os.path.join(path, 'train.csv')\n            train_df = pd.read_csv(p)\n            # Set labels\n            if not p_labels is None:\n                train_df.columns = p_labels\n            # Prepare Training, Validation and Test datasets\n            print('----------------------------- kaggle_pre_load_images_datasets: training dataset')\n            # Build full path images\n            p = os.path.join(path, 'train')\n            train_df['image_path'] = train_df['image_id'].apply(lambda x: p + '/%s' % x)\n            train_df.drop(['image_id'], inplace = True, axis = 1)\n            # Split labels into classes as descibed by IMAGE_CLASSES\n            l = _convert_multi_labels(train_df['class'], p_length = len(IMAGE_CLASSES))\n            classes = pd.DataFrame(l, columns = IMAGE_CLASSES)\n            train_df.drop(['class'], inplace = True, axis = 1)\n            train_df = pd.concat([train_df, classes], axis = 1)\n            # Read the test files and build the files list\n            print('----------------------------- kaggle_pre_load_images_datasets: test dataset')\n            if not p_test_url is None:\n                p = os.path.join(path, 'test')\n                test_files = tf.io.gfile.glob(os.path.join(p, '*'))\n                test_files = [(test_files[i].split('_')[0]) for i in range(0, len(test_files), 4)]\n                # Put them in dataset\n                test_df = pd.DataFrame(test_files, columns = ['image_path'])\n            else:\n                raise Exception('kaggle_pre_load_images_datasets', 'Test url not provided')\n        elif  ML_NAME == 'BMS-MolecularTranslation':\n            # Add a 'full path' feature\n            p = os.path.join(path, 'train_labels.csv')\n            train_df = pd.read_csv(p)\n            # Set labels\n            if not p_labels is None:\n                train_df.columns = p_labels\n            train_df['full_path'] = train_df['image_id'].apply(lambda x: p + '/%c/%c/%c/%s.png' % (x[0], x[1], x[2], x))\n            # FIXME Process test_df \n            raise Exception('kaggle_pre_load_images_datasets', 'FIXME Process test_df')\n        else:\n            raise Exception('kaggle_pre_load_images_datasets', 'Unsupported ML_NAME: %s' % ML_NAME)\n    \n        if not p_validation_url is None:\n            p = os.path.join(path, p_validation_url)\n            if  ML_NAME == 'BMS-MolecularTranslation':\n                # Add a 'full path' feature\n                p = os.path.join(path, 'validation_labels.csv')\n                validation_df = pd.read_csv(p)\n                # Set labels\n                if not p_labels is None:\n                    validation_df.columns = p_labels\n                validation_df['full_path'] = validation_df['image_id'].apply(lambda x: p + '/%c/%c/%c/%s.png' % (x[0], x[1], x[2], x))\n            else:\n                raise Exception('kaggle_pre_load_images_datasets', 'To be implemented')\n        else: # Need to extract randomly Validation datatset from Training dataset\n            train_df, validation_df = model_selection.train_test_split(train_df, train_size = p_train_size)\n            # reindex after split\n            train_df.reset_index(inplace = True)\n            validation_df.reset_index(inplace = True)\n\n        print('----------------------------- kaggle_pre_load_images_datasets: training dataset')\n        print(train_df.shape)\n        #print(validation_df.head())\n        print('----------------------------- kaggle_pre_load_images_datasets: validation dataset')\n        print(validation_df.shape)\n        #print(validation_df.head())\n        print('----------------------------- kaggle_pre_load_images_datasets: test dataset')\n        print(test_df.shape)\n        #print(test_df.head())\n    else:\n        pass\n\n    print('kaggle_pre_load_images_datasets: Done')\n    return train_df, validation_df, test_df\n\ndef kaggle_post_load_images_datasets(p_train_df, \n                                     p_validation_df, \n                                     p_train_df_num_images: int, \n                                     p_validation_df_num_images: int\n                                    ) -> list:\n    print('----------------------------- kaggle_post_load_images_datasets -----------------------------')\n    print('kaggle_post_load_images_datasets: ', type(p_train_df))\n\n    print('kaggle_post_load_images_datasets: Done')\n    return p_train_df, p_validation_df, p_train_df_num_images, p_validation_df_num_images\n\ndef kaggle_pre_features_engineering(p_df: pd.core.frame.DataFrame) -> pd.core.frame.DataFrame:\n    \"\"\"\n    Apply feature processing resulting of the data analyzing just before to start data engineering (point 3.a).\n    \"\"\"\n    print('----------------------------- kaggle_pre_features_engineering -----------------------------')\n    print(p_df.describe().T)\n\n    if ML_NAME == 'Pima': \n        # FEATURES_PROCESSING\n        print('kaggle_pre_features_engineering: Replacing impossible 0 values by NaN value')\n        p_df['plas'] = p_df['plas'].apply(lambda p_value : np.NaN if p_value == 0 else p_value)\n        p_df['pres (mm Hg)'] = p_df['pres (mm Hg)'].apply(lambda p_value : np.NaN if p_value == 0 else p_value)\n        p_df['skin (mm)'] = p_df['skin (mm)'].apply(lambda p_value : np.NaN if p_value == 0 else p_value)\n        p_df['test (mu U/ml)'] = p_df['test (mu U/ml)'].apply(lambda p_value : np.NaN if p_value == 0 else p_value)\n        p_df['mass'] = p_df['mass'].apply(lambda p_value : np.NaN if p_value == 0 else p_value)\n    elif ML_NAME == 'Titanic':\n        # S has the higher cardinality (see kaggle_summurize_data: distribution of categorical features)\n        p_df['Embarked'] = p_df['Embarked'].apply(lambda p_value : p_value[0:1] if not p_value is np.NaN else 'S')\n        print('----------------------------- kaggle_pre_features_engineering: Features creation')\n        p_df['FamilySize'] = p_df.apply(lambda p_df : p_df['SibSp'] + p_df['Parch'] + 1, axis = 1)\n        # Create class of ages based on common Age distribution, NaN values will be imputed\n        p_df['AgeClass'] = p_df.apply(lambda p_df : 'Senior' if p_df['Age'] >= 60 else 'Adult' if p_df['Age'] >= 35 else 'Young Adult' if p_df['Age'] >= 25 else 'Teen' if p_df['Age'] >= 14 else 'Child' if p_df['Age'] >= 4 else 'Baby', axis = 1)\n        # Create class of fare based on discussion below\n        p_df['FareClass'] = p_df.apply(lambda p_df : 'Very Expensive' if p_df['Fare'] >= (3*512/4) else 'Expensive' if p_df['Fare'] >= (512/2) and p_df['Fare'] < (3*512/4) else 'Chip' if p_df['Fare'] < (512/2) and p_df['Fare'] >= (512/4) else 'Very Chip', axis = 1)\n        print('----------------------------- kaggle_pre_features_engineering: Features deletion')        \n        # SibSp and Parch were repaced by FamilySize, Age by AgeClass and Fare by FareClass\n        p_df.drop(['SibSp', 'Parch', 'Age', 'Fare'], inplace = True, axis = 1)\n        print('----------------------------- kaggle_pre_features_engineering: After features creation/deletion:')\n        #print(p_df.head())\n        print(p_df.describe().T)\n\n    print('kaggle_pre_features_engineering: Done')\n    return p_df\n\ndef kaggle_post_features_engineering(p_df: pd.core.frame.DataFrame) -> pd.core.frame.DataFrame:\n    \"\"\"\n    Apply feature processing resulting of the data analyzing at the end of the data engineering (point 3.a).\n    \"\"\"\n    print('----------------------------- kaggle_post_features_engineering -----------------------------')\n    print('kaggle_post_features_engineering: Done')\n    return p_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading the Datasets/Images\n\nBefore to load and to examine our datasets (point c.1), we are just going to set a number of defaults such as the settings for the plotting operation, Deep Learning parameters... (point b.2)\n\nNotes:\n- Point b.4 provides some helpers functions to optimize Neural Networks executoin using TPUs or GPUs.  TPU stands for Tensorflow Processor Unit","metadata":{}},{"cell_type":"code","source":"# b.2) Set some defaults\ndef kaggle_set_mp_default() -> None:\n    \"\"\"\n    Some default setting for Matplotlib plots\n    \"\"\"\n    warnings.filterwarnings(\"ignore\") # to clean up output cells\n    pd.set_option('precision', 3)\n    # Set Matplotlib defaults\n    plt.rc('figure', autolayout=True)\n    plt.rc('axes', labelweight='bold', labelsize='large', titleweight='bold', titlesize=18, titlepad=10)\n    plt.rc('image', cmap='magma')\n    # End of function set_mp_default\n\n# Fix random values for reproductibility\nSEED_HARCODED_VALUE = 42\n\ndef kaggle_set_seed(p_seed: int = SEED_HARCODED_VALUE) -> None:\n    \"\"\"\n    Random reproducability\n    :parameter p_seed: Set the seed value for random functions reproductibility\n    \"\"\"\n    seed = p_seed\n    if seed is None:\n        seed = random.randint(0, 10000) \n    np.random.seed(seed)\n    sklearn.utils.check_random_state(seed)\n    tf.random.set_seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    os.environ['TF_DETERMINISTIC_OPS'] = '1'\n    # End of function set_seed\n\ndef kaggle_modules_versions() -> None:\n    \"\"\"\n    Print the different modules version\n    \"\"\"\n    print('----------------------------- modules_versions -----------------------------')\n    print(\"Numpy version: \" + np.__version__)\n    print('seaborn: %s' % sns.__version__)\n    print(\"Pandas version: \" + pd.__version__)\n    print(\"Sklearn version: \" + sklearn.__version__)\n    print(\"Tensorflow version: \" + tf.__version__)\n    print('modules_versions: Done')\n    # End of function modules_versions","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In case of working with Neural Networks, set some additional defaults and doing a TPU detection are required.","metadata":{}},{"cell_type":"code","source":"# b.4) Set additional defaults for Neural Networks\nDL_BATCH_SIZE = 16 # Default batch size for DL models \nDL_EPOCH_NUM = 40 # Default epoch number for DL models.\nDL_LEARNING_RATE = 0.002 # Default learning rate for DL models \nDL_DROP_RATE = 0.2 # Default drop rate for DL models \nDL_INPUT_SHAPE = None # Default input shape size for DL models, will be used for cross_validation (see kaggle_check_models)\n\n# Doing TPU detection\ndef kaggle_tpu_detection():\n    \"\"\"\n    This function provides a TPU detection. If TPU is not supported, a default strategy is returned.\n    Note: This method also setup the global path to access datasets when TPU (from Kaggle environment) is detected.\n    :return: The appropriate distribution strategy and the global path to access datasets (None if no TPU support)\n    \"\"\"\n    print('----------------------------- kaggle_tpu_detection -----------------------------')\n    global_path = None\n    try: \n        # Detect the TPU\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n        print('kaggle_tpu_detection: Running on TPU ', tpu.master())\n    except ValueError:\n        tpu = None\n    if tpu:\n        # Initialize the TPU\n        print(\"kaggle_tpu_detection: List of devices: \", tf.config.list_logical_devices('TPU'))\n        tf.config.experimental_connect_to_cluster(tpu)\n        tf.tpu.experimental.initialize_tpu_system(tpu)\n        # Instantiate a distribution strategy\n        strategy = tf.distribute.experimental.TPUStrategy(tpu)\n        if strategy.num_replicas_in_sync > 1:\n            # Update paths accordingly\n            from kaggle_datasets import KaggleDatasets\n            global_path = KaggleDatasets().get_gcs_path(DATABASE_NAME)\n    else:\n        strategy = tf.distribute.get_strategy() \n    print('kaggle_tpu_detection: replica=%s' % str(strategy.num_replicas_in_sync))\n    print('kaggle_tpu_detection: global_path: ', global_path)\n    \n    print('kaggle_tpu_detection Done')\n    return strategy, global_path\n    # End of function kaggle_tpu_detection\n\n# Tensorflow specific helper functions\ndef kaggle_bytes_to_tfrecord(p_value):\n    \"\"\"\n    Returns a bytes_list from a string/byte\n    \"\"\"\n    if isinstance(p_value, type(tf.constant(0))):\n        p_value = p_value.numpy()\n    return tf.train.Feature(bytes_list = tf.train.BytesList(value = [p_value]))\n    # End of function kaggle_bytes_to_tfrecord\n\ndef kaggle_floats_to_tfrecord(p_value):\n    \"\"\"\n    Returns a float_list from a float/double\n    \"\"\"\n    return tf.train.Feature(float_list = tf.train.FloatList(value = [p_value]))\n    # End of function kaggle_floats_to_tfrecord\n\ndef kaggle_ints_to_tfrecord(p_value):\n    \"\"\"\n    Returns an int64_list from a bool/enum/int/uint\n    \"\"\"\n    return tf.train.Feature(int64_list = tf.train.Int64List(value = [p_value]))\n    # End of function kaggle_ints_to_tfrecord\n\ndef kaggle_dataset_size(p_filenames: list) -> int:\n    \"\"\"\n    The number of data items in the dataset is written in the name of the .tfrec files (i.g. features00-230.tfrec = 230 data items in the file features00-230.tfrec)\n    \"\"\"\n    print('----------------------------- kaggle_dataset_size -----------------------------')\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in p_filenames]\n    print('kaggle_dataset_size: n=', n)\n    print('kaggle_dataset_size: np.sum(n)=', np.sum(n))\n    return np.sum(n)\n    # End of function kaggle_dataset_size\n\ndef kaggle_decode_png_image(p_image_data, p_channels: int = 3):\n    \"\"\"\n    This function decodes the PNG image and applies required Tensorflow adjustment\n    Note: The image was stored in a Tfrecord dataset\n    :parameter p_image_data: \n    :parameter p_channels: \n    :return: The decoded image\n    \"\"\"\n    image = tf.image.decode_png(p_image_data, channels = p_channels)\n    # Convert image to floats in [0, 1] range\n    image = tf.cast(image, tf.float32) / 255.0\n    # Explicit size needed for TPU\n    image = tf.reshape(image, [*IMAGE_SIZE, p_channels])\n    return image\n    # End of function kaggle_decode_png_image\n\ndef kaggle_decode_jpeg_image(p_image_data, p_channels: int = 3):\n    \"\"\"\n    This function decodes the JPEG image and applies required Tensorflow adjustment\n    Note: The image was stored in a Tfrecord dataset\n    :parameter p_image_data: \n    :parameter p_channels: \n    :return: The decoded image\n    \"\"\"\n    image = tf.image.decode_jpeg(p_image_data, channels = p_channels)\n    # Convert image to floats in [0, 1] range\n    image = tf.cast(image, tf.float32) / 255.0\n    # Explicit size needed for TPU\n    image = tf.reshape(image, [*IMAGE_SIZE, p_channels])\n    return image\n    # End of function kaggle_decode_jpeg_image\n\n# Tensorflow Protocol buffer for unlabeled images stroed into a Tfrecord dataset.\n# Labels are replaced by identifiers\nunlabeled_image_feature_description = {\n    'image': tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n    'id': tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n}\ndef kaggle_read_unlabeled_tfrecord(p_data):\n    data = tf.io.parse_single_example(p_data, unlabeled_image_feature_description)\n    image = kaggle_decode_jpeg_image(data['image'])\n    idnum = data['id']\n    return image, idnum # returns a dataset of image(s)\n    # End of function kaggle_read_unlabeled_tfrecord\n\n# Tensorflow Protocol buffer for labeled images stroed into a Tfrecord dataset\nlabeled_image_feature_description = {\n    'image': tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n    'class': tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n}\ndef kaggle_read_labeled_tfrecord(p_data):\n    data = tf.io.parse_single_example(p_data, labeled_image_feature_description)\n    image = kaggle_decode_jpeg_image(data['image'])\n    label = tf.cast(data['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n    # End of function kaggle_read_labeled_tfrecord\n\n# FIXME Development in progress\ndef kaggle_build_tfrec_dataset(p_train_url: str, p_validation_url: str, p_test_url: str):\n    \"\"\"\n    This function is called by kaggle_load_images_datasets() just before to start processing\n    The Images database shall be organized as follow:\n    1) In Tfrecord format\n        <current_folder>/tfrecords-jpeg-<n>x<n>/train\n        <current_folder>/tfrecords-jpeg-<n>x<n>/val\n        <current_folder>/tfrecords-jpeg-<n>x<n>/test\n    2) Otherwose:\n        <current_folder>/train_labels.csv\n        <current_folder>/train\n        [<current_folder>/train], could not exist\n        <current_folder>/test\n    \"\"\"\n    print('----------------------------- kaggle_build_tfrec_dataset -----------------------------')\n    if not p_train_url is None and p_validation_url is None and p_test_url is None:\n        if ML_NAME == 'BMS-MolecularTranslation':\n            # Convert separated Images/Labels in TFrecord format\n            path = os.path.join(os.path.abspath(os.getcwd()), '../input')\n            path = os.path.join(path, p_train_url)\n            print('kaggle_build_tfrec_dataset: path=', path)\n            # Process Training dataset\n            # 1. Load labels\n            print('kaggle_build_tfrec_dataset: Loading train labels')\n            labels_df = pd.read_csv(os.path.join(path, 'train_labels.csv'))\n            labels_df['full_path'] = labels_df['image_id'].apply(lambda x: './train/%c/%c/%c/%s.png' % (x[0], x[1], x[2], x))\n            print(labels_df.head())\n            print(labels_df.tail())\n            raise Exception('Stop')\n            # 2. Create Tfrecord dataset for training\n            print('kaggle_build_tfrec_dataset: Creating Tfrecord dataset for training')\n            path = os.path.join(path, 'train')\n            modulus = 512\n            counter = 0\n            i = 0\n            tw = tf.io.TFRecordWriter('./%.2i-%ix%i-%i.tfrec' % (counter, IMAGE_NUM_PIXELS, IMAGE_NUM_PIXELS, modulus))\n            for root, subdirs, files in os.walk(path):\n                for filename in files:\n                    file_path = os.path.join(root, filename)\n                    # load the image\n                    img = load_img(file_path, (IMAGE_NUM_PIXELS, IMAGE_NUM_PIXELS))\n                    # Convert to numpy array\n                    pixels = np.asarray(img) #<class 'numpy.ndarray'>\n                    # Confirm pixel range is 0-255\n                    #print('Data Type: %s' % pixels.dtype)\n                    #print('Min: %.3f, Max: %.3f' % (pixels.min(), pixels.max()))\n                    # Convert from integers to floats\n                    pixels = pixels.astype('float32')\n                    row = labels_df.loc[labels_df.image_id == os.path.splitext(os.path.basename(filename))[0], ['InChI']]\n                    #kaggle_display_image_and_component(pixels, filename)\n                    feature = {\n                        'image': kaggle_bytes_to_tfrecord(pixels.tobytes()),\n                        'class': kaggle_bytes_to_tfrecord(str.encode(row['InChI'].tolist()[0]))\n                    }\n                    record_bytes = tf.train.Example(features = tf.train.Features(feature = feature)).SerializeToString()\n                    tw.write(record_bytes)\n                    i += 1\n                    if i % modulus == 0:\n                        tw.close()\n                        print('kaggle_build_tfrec_dataset: Writing TFRecord %i of %i...'%(counter, i))\n                        i = 0 # Reset i\n                        counter += 1\n                        tw = tf.io.TFRecordWriter('./%.2i-%ix%i-%i.tfrec' % (counter, IMAGE_NUM_PIXELS, IMAGE_NUM_PIXELS, modulus))\n                    # End of 'for' statement\n                # End of 'for' statement\n            print('kaggle_build_tfrec_dataset: counter=%d, i=%d' % (counter, i))\n            tw.close()\n            if (i % modulus != 0):\n                os.rename(\n                    './%.2i-%ix%i-%i.tfrec' % (counter, IMAGE_NUM_PIXELS, IMAGE_NUM_PIXELS, modulus),\n                    './%.2i-%ix%i-%i.tfrec' % (counter, IMAGE_NUM_PIXELS, IMAGE_NUM_PIXELS, i)\n                         )\n            print('kaggle_build_tfrec_dataset: frecord dataset for training: Done')\n            # 3. Create Tfrecord dataset for test\n            print('kaggle_build_tfrec_dataset: Creating Tfrecord dataset for test')\n\n            print('kaggle_build_tfrec_dataset: Tfrecord dataset for test: Done')\n            # End of 'if' statement, ML_NAME == 'BMS-MolecularTranslation'\n        raise Exception('Stop')\n    else:\n        raise Exception('kaggle_build_tfrec_dataset', 'Wrong parameters')\n    print('kaggle_build_tfrec_dataset: Done')\n    # End of function kaggle_build_tfrec_dataset","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we are ready to load our dataset and examine it to understand the data it contains. This function accept any URI (e.g. file:///... or http://... or https://...).\n\nLoading the dataset, you can specify or overwrite columns labels.\n\nAccording to the data analyzing, you can also define some post loading processing using lambda function (see kaggle_post_load_datasets).\n\nThe function kaggle_load_datasets() splits the data into three datasets:\n- Training dataset used to train the model(size fixed by p_train_size, default is 90%)\n- Test dataset use to test the mode with unseen data (size is (100 - p_train_size), default is 10%)\n- Training dataset is splitted again into Training dataset (80%) and  Validation dataset used to fit the model (size is 20%)\n\nNote: The Test dataset does not contain target features (see TARGET_COLUMNS)","metadata":{}},{"cell_type":"code","source":"# c.1) Load 'Data' dataset\ndef kaggle_load_datasets(p_url: str, \n                         p_labels: list = None, \n                         p_train_path: str = None, \n                         p_validation_path: str = None,\n                         p_train_size: float = 0.9,\n                         p_seed: int = SEED_HARCODED_VALUE\n                        ) -> list:\n    \"\"\"\n    This function load the dataset specified by p_url or (p_train_path, p_validation_path) ina case of Kaggle compete.\n    It also add the labels if required and apply post load processing of the datatsets if required\n    :parameters p_url: The URI of the dataset (http:// or file://)\n    :parameters p_labels: The label of the columns to be used. Default: None\n    :parameters p_train_path: Kaggle specific path for train dataset\n    :parameters p_validation_path: Kaggle specific path for validation dataset\n    :parameter p_seed: The seed value for reproductibility\n    :return: Four datasets: The Training, Validation and Test datasets. The Test dataset does no contain the outputs, it acts as unseen data for the model. This is the fourth dataset returned\n    \n    :exception: Raised if specified link is not correct\n    \"\"\"\n    print('----------------------------- kaggle_load_datasets -----------------------------')\n    train_df = None\n    validation_df = None \n    test_df = None\n    y_test_df = None\n    \n    kaggle_pre_load_datasets()\n\n    if not p_train_path is None and not p_validation_path is None:\n        # Kaggle compete specific\n        train_df = pd.read_csv(p_train_path)\n        test_df = pd.read_csv(p_validation_path) # y_test_df will be None for the Kaggle compete\n        # Set labels\n        if not p_labels is None:\n            df.columns = p_labels\n        # Split train_df into Training and Test datasets\n        y_train_df = train_df[TARGET_COLUMNS]\n        x_train_df = train_df.drop([TARGET_COLUMNS], axis = 1)\n        X_train, X_validation, Y_train, Y_validation = model_selection.train_test_split(x_train_df, y_train_df, train_size = 0.8, random_state = p_seed)               \n        train_df = pd.concat([X_train, Y_train], axis = 1)\n        validation_df = pd.concat([X_validation, Y_validation], axis = 1)       \n    else:\n        # Get the data\n        if p_url.startswith('file://'):\n            df = pd.read_csv(p_url[7:])\n        elif p_url.startswith('http'):\n            ds = requests.get(p_url).content\n            df = pd.read_csv(io.StringIO(ds.decode('utf-8')))\n        if df is None:\n            raise Exception('kaggle_load_datasets: Failed to load data frame', 'url=%s' % (url))\n        # Set labels\n        if not p_labels is None:\n            df.columns = p_labels\n        # Split them into Training, Test and Validation datasets\n        y_df = df[TARGET_COLUMNS]\n        x_df = df.drop([TARGET_COLUMNS], axis = 1)\n        X_train, X_test, Y_train, Y_test = model_selection.train_test_split(x_df, y_df, train_size = p_train_size, random_state = p_seed)\n        train_df = pd.concat([X_train, Y_train], axis = 1)\n        test_df = X_test\n        y_test_df = Y_test\n\n        y_df = train_df[TARGET_COLUMNS]\n        x_df = train_df.drop([TARGET_COLUMNS], axis = 1)\n        X_train, X_validation, Y_train, Y_validation = model_selection.train_test_split(x_df, y_df, train_size = 0.8, random_state = p_seed)\n        train_df = pd.concat([X_train, Y_train], axis = 1)\n        validation_df = pd.concat([X_validation, Y_validation], axis = 1)\n\n    #print('----------------------------- kaggle_load_datasets: training dataset')\n    #print(train_df.describe().T)\n    #print('----------------------------- kaggle_load_datasets: validation dataset')\n    #print(validation_df.describe().T)\n    #print('----------------------------- kaggle_load_datasets: test dataset')\n    #print(test_df.describe().T)\n    \n    # Apply post processing after loading dataset\n    train_df, validation_df, test_df = kaggle_post_load_datasets(train_df, validation_df, test_df)\n\n    print('----------------------------- kaggle_load_datasets: training dataset')\n    print(train_df.head())\n    print('----------------------------- kaggle_load_datasets: validation dataset')\n    print(validation_df.head())\n    print('----------------------------- kaggle_load_datasets: test dataset')\n    print(test_df.head())\n\n    print('kaggle_load_datasets: Done: %s' % (p_url if not p_url is None else p_train_path))\n    return train_df, validation_df, test_df, y_test_df\n    # End of function kaggle_load_datasets","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The function bellow loads images dataset. The images shall be store in one of the following format:\n- Separated images/labels with one of the following format:\n1. One folder for each of Training, Validation and Test datasets. Each folder contains images in PNG or JPEG format and the labels in in a CSV file\n2. TODO...\n- TensorFlow 'tfrec' format, with or without label defined.","metadata":{}},{"cell_type":"code","source":"# c.2) Load 'Images' dataset\ndef kaggle_load_images_datasets(p_train_url: str, \n                                p_labels: list = None,\n                                p_global_path: str = None,\n                                p_validation_url: str = None,\n                                p_test_url: str = None, \n                                p_train_path: str = None, \n                                p_validation_path: str = None,\n                                p_test_path: str = None,\n                                p_train_size: float = 0.9,\n                                p_ordered: bool = False,\n                                p_seed: int = SEED_HARCODED_VALUE\n                                ) -> list:\n    \"\"\"\n    This function loads Tensorflow Tfrecord datasets.\n    \n    :parameters p_labels: The label of the columns to be used. Default: None\n    \n    \n    :parameter p_train_path: Path of the Training 'Tfrec' folder  \n    :parameter p_validation_path: Path of the Validation 'Tfrec' folder\n    :parameter p_test_path: Path of the Test 'Tfrec' folder\n    :parameter p_train_size: Size ratio of Training dataset vs. Validation dataset\n    :parameter p_seed: The seed value for reproductibility\n    :return: Four datasets: The Training, Test and Validation datasets. The Test dataset does no contain the outputs, it acts as unseen data for the model. This is the fourth dataset returned\n    \"\"\"\n    print('----------------------------- kaggle_load_images_datasets -----------------------------')\n\n    train_df, validation_df, test_df = kaggle_pre_load_images_datasets(p_train_url = p_train_url, \n                                                                       p_labels = p_labels, \n                                                                       p_global_path = p_global_path,\n                                                                       p_validation_url = p_validation_url, \n                                                                       p_test_url = p_test_url,\n                                                                       p_train_size = p_train_size\n                                                                       )\n    #print('----------------------------- kaggle_load_images_datasets: training dataset')\n    #print(train_df.head())\n    #print('----------------------------- kaggle_load_images_datasets: validation dataset')\n    #print(validation_df.head())\n    #print('----------------------------- kaggle_load_images_datasets: test dataset')\n    #print(test_df.head())\n\n    train_df_num_images = None\n    validation_df_num_images = None\n    test_df_num_images = None\n    if not train_df is None:\n        print('kaggle_load_images_datasets: Processing compete %s:' % ML_NAME)\n        if ML_NAME == 'HumanProteinAtlas':\n            def _rebuild_image_from_channels(p_image_path: str, p_label: list = None, p_seed: int = SEED_HARCODED_VALUE) -> list:\n                \"\"\"\n                This function convert image channels into one RGB image\n                \"\"\"\n                red = tf.io.read_file(p_image_path + \"_red.png\")\n                blue = tf.io.read_file(p_image_path + \"_blue.png\")\n                green = tf.io.read_file(p_image_path + \"_green.png\")\n                yellow = tf.io.read_file(p_image_path + \"_yellow.png\")\n\n                red = tf.io.decode_png(red, channels = 1) # Grayscale image\n                blue = tf.io.decode_png(blue, channels = 1) # Grayscale image\n                green = tf.io.decode_png(green, channels = 1) # Grayscale image\n                yellow = tf.io.decode_png(yellow, channels = 1) # Grayscale image\n                \n                red = tf.math.maximum(red, yellow)\n                blue = tf.math.maximum(blue, yellow)\n\n                # Convert image to floats in [0, 1] range\n                red = tf.cast(red, tf.float32) / 255.0\n                blue = tf.cast(blue, tf.float32) / 255.0\n                green = tf.cast(green, tf.float32) / 255.0\n                \n                # Explicit size needed for TPU\n                red = tf.image.resize(red, [*IMAGE_SIZE])\n                blue = tf.image.resize(blue, [*IMAGE_SIZE])\n                green = tf.image.resize(green, [*IMAGE_SIZE])\n\n                red = tf.squeeze(red)\n                blue = tf.squeeze(blue)\n                green = tf.squeeze(green)\n                \n                # Build RGB image, channels last: shape is (width, high, channel)\n                # In this case, shape is (1024, 1024, 3):\n                #     1024 entries of (1024 lines per 3 columns)\n                #         First entry, first line:   [ R[0,0], G[0,0], B[0,0] ]\n                #         First entry, second line:  [ R[0,1], G[0,1], B[0,1] ]\n                #         ...\n                #         Second entry, first line:  [ R[1,0], G[1,0], B[1,0] ]\n                #         ....\n                # Stack structure is: each entry of the stack is the tuple (R, G, B) of  \n                # Axis value: 2 for channels last, -1 for channel first\n                # From stack, to extract one channel (R, G or B): stack[:, :, n], n = 0 for R, 1, for G and 2 for B\n                image = tf.stack([red, green, blue], axis = 2) # RGB channels last\n\n                return image, p_label\n                # End of _rebuild_image_from_channels\n\n            # Sample elements from Tests and Validation dataset to reduce execution time\n            if not IMAGES_SAMPLES_NUM is None: # Use a subset of images from the training Dataset\n                # Shuffle row from the Training dataset\n                train_df = train_df.sample(n = IMAGES_SAMPLES_NUM, random_state = p_seed)\n                train_df.reset_index(inplace = True);\n                # Shuffle row from the Validation dataset\n                validation_df = validation_df.sample(n = int((1 - p_train_size) * IMAGES_SAMPLES_NUM), random_state = p_seed)\n                validation_df.reset_index(inplace = True);\n            else: # Use all images from the training Dataset\n                # Nothing to do\n                pass\n            # Set sizes\n            train_df_num_images = train_df.shape[0]\n            validation_df_num_images = validation_df.shape[0]\n            test_df_num_images = test_df.shape[0]\n            # Build Tensoflow dataset for Training\n            image_path = train_df['image_path'] # List of image paths as pd.Series\n            labels = train_df[IMAGE_CLASSES] # List of labels as pd.DataFrame\n            print('Training image_path shape ==> ', image_path.shape)\n            print('Training labels shape ==> ', labels.shape)\n            # Conver Series/DataFrame into np.darray before to convert them into tensors\n            image_path = np.array(image_path.values.tolist())\n            #print(type(image_path))\n            #print(image_path)\n            labels = np.array(labels.values.tolist())\n            #print(type(labels))\n            #print(labels)\n            # Testing _rebuild_image_from_channels function\n            #i, t = _rebuild_image_from_channels(image_path[0], labels[0])\n            #print(i.shape)\n            #print(i)\n            #print(t)\n            #raise Exception('Stop')\n            # Convert them into tensors\n            train_df = tf.data.Dataset.from_tensor_slices((image_path, labels))\n            # Shuffle the training dataset\n#            train_df = train_df.shuffle(len(image_path) + 1)  # buffer_size >= dataset length\n            train_df = train_df.map(_rebuild_image_from_channels, num_parallel_calls = tf.data.AUTOTUNE)\n            # Inspect Training dataset\n            for image, label in train_df.take(3):\n                print('kaggle_load_images_datasets: Training data label inspection:', label.numpy())\n                print(image.numpy().shape, len(image.numpy()), label.numpy().shape)\n            # Build Tensoflow dataset for Validation\n            image_path = validation_df['image_path'] # List of image paths as pd.Series\n            labels = validation_df[IMAGE_CLASSES] # List of labels as pd.DataFrame\n            print('Validation image_path shape ==> ', image_path.shape)\n            print('Validation labels shape == ', labels.shape)\n            # Conver Series/DataFrame into np.darray before to convert them into tensors\n            image_path = np.array(image_path.values.tolist())\n            #print(type(image_path))\n            #print(image_path)\n            labels = np.array(labels.values.tolist())\n            #print(type(labels))\n            #print(labels)\n            # Convert them into tensors\n            validation_df = tf.data.Dataset.from_tensor_slices((image_path, labels))\n            # Shuffle the validation dataset\n#            validation_df = validation_df.shuffle(len(image_path) + 1)  # buffer_size >= dataset length\n            validation_df = validation_df.map(_rebuild_image_from_channels, num_parallel_calls = tf.data.AUTOTUNE)\n            # Inspect Validation dataset\n            for image, label in validation_df.take(3):\n                print('kaggle_load_images_datasets: Validation data label inspection:', label.numpy())\n                print(image.numpy().shape, len(image.numpy()), label.numpy().shape)\n            # Build Tensoflow dataset for Validation\n            image_path = test_df['image_path'] # List of image paths\n            # Conver Series/DataFrame into np.darray before to convert them into tensors\n            image_path = np.array(image_path.values.tolist())\n            #print('type(image_path) ==> ', type(image_path))\n            #print(image_path)\n            test_df = tf.data.Dataset.from_tensor_slices(image_path)\n            # Shuffle the test dataset\n#            test_df = test_df.shuffle(len(image_path) + 1)  # buffer_size >= dataset length\n            test_df = test_df.map(_rebuild_image_from_channels, num_parallel_calls = tf.data.AUTOTUNE)\n            # Inspect Validation dataset\n            for image, label in validation_df.take(3):\n                print('kaggle_load_images_datasets: Test data inspection:')\n                print(image.numpy().shape, len(image.numpy()))\n        #elif ML_NAME == 'BMS-MolecularTranslation':\n        #    pass\n        else:\n            raise Exception('kaggle_load_images_datasets', 'Unsupported ML_NAME: %s' % ML_NAME)\n    elif ML_NAME == 'FlowerClassification':\n        print('kaggle_load_images_datasets: Processing compete %s:' % ML_NAME)\n        # Load file names for Train, Validation and Test images folders\n        train_files = tf.io.gfile.glob(os.path.join(p_global_path, p_train_path))\n        # Sample elements from Tests and Validation dataset to reduce execution time\n        # FIXME How to sample Tensorflow datasets?\n        #if not IMAGES_SAMPLES_NUM is None: # Use a subset of images from the training Dataset\n        #    # Shuffle row from the Training dataset\n        #    train_files = random.sample(train_files, IMAGES_SAMPLES_NUM)\n        #    train_files.reset_index(inplace = True);\n        #else: # Use all images from the training Dataset\n        #    # Nothing to do\n        #    pass\n        # Disabling order increases speed\n        ignore_order = tf.data.Options()\n        if not p_ordered:\n            ignore_order.experimental_deterministic = False # disable order, increase speed\n        # Build the dataset with the images\n        train_df = tf.data.TFRecordDataset(train_files, num_parallel_reads = tf.data.experimental.AUTOTUNE)\n        # To use data as soon as it streams in\n        train_df = train_df.with_options(ignore_order)\n        # Decode tfrecord images into (jpeg, label)\n        train_df = train_df.map(kaggle_read_labeled_tfrecord, num_parallel_calls = tf.data.experimental.AUTOTUNE)\n        print('kaggle_load_images_datasets: Training data shapes:', train_df.cardinality().numpy())\n        # Inspect Training dataset\n        for image, label in train_df.take(3):\n            print('kaggle_load_images_datasets: Training data label inspection:', label.numpy())\n            print(image.numpy().shape, len(image.numpy()), label.numpy().shape)\n        train_df_num_images = kaggle_dataset_size(train_files)\n        if not p_validation_path is None:\n            validation_files = tf.io.gfile.glob(os.path.join(p_global_path, p_validation_path))\n            # Sample elements from Tests and Validation dataset to reduce execution time\n            # FIXME How to sample Tensorflow datasets?\n            #if not IMAGES_SAMPLES_NUM is None: # Use a subset of images from the training Dataset\n            #    # Shuffle row from the Validation dataset\n            #    validation_df = random.sample(validation_df, int((1 - p_train_size) * IMAGES_SAMPLES_NUM // p_train_size))\n            #    validation_df.reset_index(inplace = True);\n            #else: # Use all images from the validation Dataset\n            #    # Nothing to do\n            #    pass\n            # Build the dataset with the images\n            validation_df = tf.data.TFRecordDataset(validation_files, num_parallel_reads = tf.data.experimental.AUTOTUNE)\n            # To use data as soon as it streams in\n            validation_df = validation_df.with_options(ignore_order)\n            # Decode tfrecord images into (jpeg, label)\n            validation_df = validation_df.map(kaggle_read_labeled_tfrecord, num_parallel_calls = tf.data.experimental.AUTOTUNE)\n            print('kaggle_load_images_datasets: Validation data shapes:', validation_df.cardinality().numpy())\n            # Inspect Validation dataset\n            for image, label in validation_df.take(3):\n                print('kaggle_load_images_datasets: Validation data label inspection:', label.numpy())\n                print(image.numpy().shape, len(image.numpy()), label.numpy().shape)\n            validation_df_num_images = kaggle_dataset_size(validation_files)\n        if not p_test_path is None:\n            test_files = tf.io.gfile.glob(os.path.join(p_global_path, p_test_path))\n            print('----------------------------- kaggle_load_images_datasets: test_files')\n            # Build the dataset with the images\n            test_df = tf.data.TFRecordDataset(test_files, num_parallel_reads = tf.data.experimental.AUTOTUNE)\n            # To use data as soon as it streams in\n            test_df = test_df.with_options(ignore_order)\n            # Decode tfrecord images into (jpeg, label)\n            test_df = test_df.map(kaggle_read_unlabeled_tfrecord, num_parallel_calls = tf.data.experimental.AUTOTUNE)\n            # Inspect Test dataset\n            for image in test_df.take(3):\n                print('kaggle_load_images_datasets: Test data inspection:')\n                print(image.numpy().shape, len(image.numpy()))\n            print('kaggle_load_images_datasets: Test data shapes:', test_df.cardinality().numpy())\n            test_df_num_images = kaggle_dataset_size(test_files)\n\n    train_df, validation_df, train_df_num_images, validation_df_num_images = kaggle_post_load_images_datasets(train_df, validation_df, train_df_num_images, validation_df_num_images)\n\n    print('kaggle_load_images_datasets: Done: %s' % (p_train_url if not p_train_url is None else p_train_path))\n    return train_df, validation_df, test_df, train_df_num_images, validation_df_num_images, test_df_num_images\n    # End of function kaggle_load_images_datasets","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Learning from data\n\nExamining the dataset means get a global overview of its data from statistical point of view, using:\n1. Some basics statistical tools such as means, stds, quartiles, skewness and correlation (2.a)\n2. Some visualization tools such as histograms, density plots (point 2.b.1) or Images display (point 2.b.2)\n\nSome additional helpers functions are also provided.\n\n\nUnderstanding the data is the most important step. The kaggle_summurize_data() function provides you a lot of information to help you in this task:\n- Dataset info: It provides information about the structure of the data:\n1) The number of features (or attributes or columns), and the name (or label) of each. Here, it is important to understand what each feature means, what can be the values for this feature, take care of the units... A lot of research work to understand our problem,\n2) The types of each feature. 'object' type indicates categorical features, it means we should have to do some imputations,\n3) One or several of these feature will be our ML output and some of them could be removed later because of poor interest to solve our problem (e.g. features with huge correlation, feature reduction using ACP...),\n3) The number of observations (or samples) in the dataset. This will be useful to split our datatset into training, validation and test dataset.\n- Dataset columns labels: It indicates the name (or label) of each attributes\n- Means: It provides you the mean value for each features (also provided by statistical abstract, see below)\n- Dataset statistical abstract: It provides, for each feature, basic statistical metrics such as means, stds, quartiles...\n- Dataset Head: It displays the fisrt samples of the dataset. It provides you some indication of the value of each observation. Note that it is not suffisient to detect specific values such as NULL or NaN values, zeros, string values, categorical values... \n- Unique possible columns: It provides, for each feature, the list of the unique values. This will help you during the data transformation to rescale and center the feature values (see point 3.c). Very often, a feature with few unique values (e.g. 2 or 3) indicates also a categorical fetaure,\n- Correlation table: It provides the correlation between all couple of features and the list of the correlation values in the range > 0.7 and < -0.7. The will be used to reduce the number of features due to strong link between some features (see p_correlation_threshold parameter)\n- Skewness Reduction: It provides, for each feature, indicators about [Skewness and Kurtosis](https://codeburst.io/2-important-statistics-terms-you-need-to-know-in-data-science-skewness-and-kurtosis-388fef94eeaa) \n\nNote: Here we use pandas_profiling to generate an analyze report in HTML format. This report is higly valuable because of the information it provides for each columns:\n1. Specific value indicators such as zeros, NaN...\n2. Distincts values\n3. Statistical values such as mean, min/max...","metadata":{}},{"cell_type":"code","source":"# 2. Summarize the dataset content in a statistical form\n# a) Descriptive statistics\ndef kaggle_summurize_data(p_df: pd.core.frame.DataFrame, p_correlation_threshold: float = 0.7) -> None:\n    \"\"\"\n    This function provides a statistical view of the current dataset\n    :parameters p_df: The dataset handle\n    \"\"\"\n    print('----------------------------- kaggle_summurize_data -----------------------------')\n    # General information\n    print('Dataset info:')\n    print(p_df.info())\n    print('----------------------------- kaggle_summurize_data: Dataset columns labels:')\n    print(p_df.columns)\n    print('----------------------------- kaggle_summurize_data: Means:')\n    print(p_df.mean())\n    print('----------------------------- kaggle_summurize_data: Dataset statistical abstract:')\n    print(p_df.describe().T)\n    print('----------------------------- kaggle_summurize_data: Dataset Head:')\n    print(p_df.head(20))\n    # NaN values\n    print('----------------------------- kaggle_summurize_data: NaN values distribution:')\n    print(p_df.isnull().sum().sort_values(ascending = False))\n    print(\"----------------------------- kaggle_summurize_data: Number of rows with NaN: \", p_df.isnull().any(axis = 1).sum())\n    # Zeros per columns\n    print('----------------------------- kaggle_summurize_data: Zeros per columns distribution:')\n    for column in p_df.columns:\n        if p_df[column].dtype == 'int64' or p_df[column].dtype == 'float64':\n            zeros = p_df[column].isin([0]).sum()\n            s = p_df[column].sum()\n            print('{}: {}'.format(column, zeros, 100 * zeros / s))\n        else:\n            print('%s: Not numerical column' % column)\n    # End of 'for' statement\n    # Distribution of categorical features\n    print('----------------------------- kaggle_summurize_data: Distribution of categorical features:')\n    categorical_columns = [col for col in p_df.columns if p_df[col].dtype == 'object']\n    for c in categorical_columns:\n        print('Distribution  for %s' % c)\n        print(p_df[c].describe())\n    # End of 'for' statement\n    # Distribution of categorical features\n    print('----------------------------- kaggle_summurize_data: Distribution of numerical features:')\n    numerical_columns = [col for col in p_df.columns if p_df[col].dtype == 'int64' or p_df[col].dtype == 'float64']\n    for c in numerical_columns:\n        print('Distribution  for %s' % c)\n        print(p_df[c].describe())\n    # End of 'for' statement\n    #  Calculate skew and kurt \n    print('----------------------------- kaggle_summurize_data: Skewness Reduction')\n    for c in numerical_columns:\n        print('Skew of %s: %f' % (c, p_df[c].skew()))\n        print('Kurt of %s: %f' % (c, p_df[c].kurt()))\n    # End of 'for' statement\n    #  Unique possible columns\n    print('----------------------------- kaggle_summurize_data: Unique possible columns:')\n    for c in p_df.columns:\n        print('{}: {}'.format(c, len(p_df[c].unique())))\n    # End of 'for' statement\n    # Build Correlation matrix\n    print('----------------------------- kaggle_summurize_data: Correlation table:')\n    print(p_df.corr(method = 'pearson'))\n    # Extract correlation > 0.7 and < -0.7\n    print('----------------------------- kaggle_summurize_data: Correlations in range > %f and < -%f:' % (p_correlation_threshold, p_correlation_threshold))\n    corr = p_df.corr().unstack().reset_index() # Group together pairwise\n    corr.columns = ['var1', 'var2', 'corr'] # Rename columns to something readable\n    print(corr[ (corr['corr'].abs() > p_correlation_threshold) & (corr['var1'] != corr['var2']) ] )\n    # Finally, create Pandas Profiling\n    #print('----------------------------- kaggle_summurize_data: Pandas Profiling:')\n    #file = ProfileReport(p_df) # Need to many times\n    #file.to_file('./eda.html')\n    #file.to_notebook_iframe()\n    print('kaggle_summurize_data: Done')\n    # End of function kaggle_summurize_data","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The functions below are some helpers for data visualization (see b.1) Data visualizations for more details).","metadata":{}},{"cell_type":"code","source":"def create_grid(p_df: pd.core.frame.DataFrame, p_features:list = None, p_nun_plot_per_line:int = 3) -> list:\n    \"\"\"\n    Create the grid in preparation of the plots\n    :parameters p_df: The dataset handle\n    :parameters p_features: The features to concider for the plot. Default: None, all the features are considered\n    :parameters p_nun_plot_per_lane: The number of plot per line. Default: 3\n    \"\"\"\n    # Create figure\n    if p_features is None:\n        features = p_df.columns\n    else:\n        features = p_features\n    sns.set_style('darkgrid')\n    l = len(features) // p_nun_plot_per_line + (1 if len(features) % p_nun_plot_per_line != 0 else 0)\n    fig = plt.figure(figsize = (l * 5, p_nun_plot_per_line * 5))\n    gs = fig.add_gridspec(l, p_nun_plot_per_line)\n    gs.update(wspace = 0.1, hspace = 0.4)\n    background_color = '#fbfbfb'\n    # Prepare the grid\n    fig_desc = dict()\n    run_no = 0\n    for row in range(0, l):\n        for col in range(0, p_nun_plot_per_line):\n            fig_desc['ax' + str(run_no)] = fig.add_subplot(gs[row, col])\n            fig_desc['ax' + str(run_no)].set_facecolor(background_color)\n            fig_desc['ax' + str(run_no)].tick_params(axis = 'y', left = True)\n            fig_desc['ax' + str(run_no)].get_yaxis().set_visible(True)\n            for s in ['top', 'right']:\n                fig_desc['ax' + str(run_no)].spines[s].set_visible(False)\n            run_no += 1\n        # End of 'for' statement\n    # End of 'for' statement\n    \n    return (fig, gs, fig_desc)\n    # End of function create_grid\n\ndef finalize_grid(p_figure_desc: list, p_df: pd.core.frame.DataFrame, p_features:list = None, p_title:str = None, p_comment:str = None) -> None:\n    \"\"\"\n    Finalize the grid after the plot\n    :parameters p_df: The dataset handle\n    :parameters p_features: The features to concider for the plot. Default: None, all the features are considered\n    :parameters p_title: The title of the figure\n    :parameters p_comment: An additional comment to add to the figure\n    \"\"\"\n    if p_features is None:\n        features = p_df.columns\n    else:\n        features = p_features\n    fig, gs, fig_desc = p_figure_desc\n    # Add Titles\n    fig_desc['ax0'].text(-0.2, 0.4, p_title, fontsize = 20, fontweight = 'bold', fontfamily = 'serif')\n    fig_desc['ax0'].text(-0.2, 0.3, p_comment, fontsize = 13, fontweight = 'light', fontfamily = 'serif')\n    # Cleanup unused plots\n    for t in range(len(features), len(fig_desc)):\n        for s in ['top', 'bottom', 'right', 'left']:\n            fig_desc['ax' + str(t)].spines[s].set_visible(False)\n        fig_desc['ax' + str(t)].tick_params(axis='x', bottom = False)\n        fig_desc['ax' + str(t)].get_xaxis().set_visible(False)\n        # End of 'for' statement\n\n    plt.show()\n\n    fig = None\n    gs = None\n    fig_desc = None\n    # End of function finalize_grid\n\ndef show_counts(p_df: pd.core.frame.DataFrame, p_features:list = None, p_hue:str = None, p_title:str = None, p_comment:str = None) -> None:\n    \"\"\"\n    \"\"\"\n    print('----------------------------- show_counts -----------------------------')\n    if p_features is None:\n        features = p_df.columns\n    else:\n        features = p_features\n    # Create the grid\n    fig, gs, fig_desc = create_grid(p_df[features])\n    # Draw plots\n    run_no = 0\n    for feature in features:\n        sns.countplot(p_df[feature], hue = p_hue, ax = fig_desc['ax' + str(run_no)], color='#ffd100')\n        run_no += 1\n    # End of 'for' statement\n    # Finalyze the figure\n    finalize_grid((fig, gs, fig_desc), p_df, features, p_title, p_comment)\n    # End of function show_distributions\n\ndef show_modes(p_df: pd.core.frame.DataFrame, p_features:list = None, p_title:str = None, p_comment:str = None) -> None:\n    print('----------------------------- show_modes -----------------------------')\n    if p_features is None:\n        features = p_df.columns\n    else:\n        features = p_features\n    # Create the grid\n    fig, gs, fig_desc = create_grid(p_df[features])\n    # Draw plots\n    run_no = 0\n    for feature in features:\n        try:\n            sns.distplot(p_df.loc[:,feature], ax = fig_desc['ax' + str(run_no)], hist = False, color='#ffd100')\n        except RuntimeError:\n            sns.distplot(p_df.loc[:,feature], ax = fig_desc['ax' + str(run_no)], kde = False, hist = False, color='#ffd100')            \n        run_no += 1\n    # End of 'for' statement\n    # Finalyze the figure\n    finalize_grid((fig, gs, fig_desc), p_df, features, p_title, p_comment)\n    # End of function show_modes\n\ndef show_distributions(p_df: pd.core.frame.DataFrame, p_features:list = None, p_title:str = None, p_comment:str = None) -> None:\n    print('----------------------------- show_distributions -----------------------------')\n    if p_features is None:\n        features = p_df.columns\n    else:\n        features = p_features\n    # Create the grid\n    fig, gs, fig_desc = create_grid(p_df[features])\n    # Draw plots\n    run_no = 0\n    for feature in features:\n        try:\n            sns.distplot(p_df.loc[:,feature], ax = fig_desc['ax' + str(run_no)], color='#ffd100')\n        except RuntimeError:\n            sns.distplot(p_df.loc[:,feature], ax = fig_desc['ax' + str(run_no)], kde = False, color='#ffd100')    \n        run_no += 1\n    # End of 'for' statement\n    # Finalyze the figure\n    finalize_grid((fig, gs, fig_desc), p_df, features, p_title, p_comment)\n    # End of function show_distributions\n\ndef show_scatter_plots(p_df: pd.core.frame.DataFrame, p_features:list = None, p_title:str = None, p_comment:str = None) -> None:\n    print('----------------------------- show_scatter_plots -----------------------------')\n    if p_features is None:\n        features = p_df.columns\n    else:\n        features = p_features\n    # Create the grid\n    fig, gs, fig_desc = create_grid(p_df[features])\n    # Draw plots\n    run_no = 0\n    for feature in features:\n        if p_df[feature].dtype == 'object':\n            sns.catplot(x = feature, data = p_df, kind = 'count', ax = fig_desc['ax' + str(run_no)], color='#ffd100')\n        else:\n            sns.catplot(x = feature, data = p_df, kind = 'swarm', ax = fig_desc['ax' + str(run_no)], color='#ffd100')\n        run_no += 1\n    # End of 'for' statement\n    # Finalyze the figure\n    finalize_grid((fig, gs, fig_desc), p_df, features, p_title, p_comment)\n    # End of function show_scatter_plots\n\ndef show_trends(p_df: pd.core.frame.DataFrame, p_features:list = None, p_title:str = None, p_comment:str = None) -> None:\n    print('----------------------------- show_trends -----------------------------')\n    if p_features is None:\n        features = p_df.columns\n    else:\n        features = p_features\n    # Create the grid\n    fig, gs, fig_desc = create_grid(p_df[features])\n    # Draw plots\n    run_no = 0\n    for feature in features:\n        sns.lineplot(data = p_df[feature], ax = fig_desc['ax' + str(run_no)], color='#ffd100')\n        run_no += 1\n    # End of 'for' statement\n    # Finalyze the figure\n    finalize_grid((fig, gs, fig_desc), p_df, features, p_title, p_comment)\n    # End of function show_trends\n\ndef show_correlations(p_df: pd.core.frame.DataFrame, p_title:str = None, p_comment:str = None) -> None:\n    print('----------------------------- show_correlations -----------------------------')\n    # Create the grid\n    fig = plt.figure(figsize = (10, 10))\n    gs = fig.add_gridspec(1, 1)\n    background_color = \"#fbfbfb\"\n    # Prepare the grid\n    fig_desc = dict()\n    fig_desc['ax0'] = fig.add_subplot(gs[0, 0])\n    fig_desc['ax0'].set_facecolor(background_color)\n    fig_desc['ax0'].tick_params(axis = 'y', left=False)\n    fig_desc['ax0'].get_yaxis().set_visible(False)\n    for s in [\"top\", \"right\", \"left\"]:\n        fig_desc['ax0'].spines[s].set_visible(False)\n    # Draw plots\n    sns.heatmap(data = p_df.corr(), annot=True)\n    # Finalyze the figure\n    # Add Titles & Comments\n    if not p_title is None:\n        fig_desc['ax0'].text(-0.2, 0.4, p_title, fontsize = 20, fontweight = 'bold', fontfamily = 'serif')\n    if not p_comment is None:\n        fig_desc['ax0'].text(-0.2, 0.3, p_comment, fontsize = 13, fontweight = 'light', fontfamily = 'serif')   \n    plt.show()\n    # End of function show_correlations\n\ndef show_outliers(p_df: pd.core.frame.DataFrame, p_title:str = None, p_comment:str = None) -> None:\n    print('----------------------------- show_outliers -----------------------------')\n    features = p_df.columns\n    # Create the grid\n    fig, gs, fig_desc = create_grid(p_df[features])\n    # Draw plots\n    run_no = 0\n    for feature in features:\n        ds = p_df[feature].value_counts()\n        sns.boxplot(ds, ax = fig_desc['ax' + str(run_no)], color='#ffd100')\n        run_no += 1\n    # End of 'for' statement\n    # Finalyze the figure\n    finalize_grid((fig, gs, fig_desc), p_df, features, p_title, p_comment)\n    # End of function show_outliers\n\ndef show_features_vs_target(p_df: pd.core.frame.DataFrame, p_target:str, p_features:list = None, p_title:str = None, p_comment:str = None) -> None:\n    print('----------------------------- show_features_vs_target -----------------------------')\n    if p_features is None:\n        features = p_df.columns.tolist() # Using tolist() for removing p_target\n    else:\n        features = p_features\n    if p_target in features:\n        features.remove(p_target)\n    # Draw plots\n    for feature in features:\n        sns.relplot(x = p_target, y = feature, data = p_df, col = p_target, color='#ffd100')\n        # End of 'for' statement\n    # End of function show_features_vs_target\n\ndef show_pair_plot_vs_target(p_df: pd.core.frame.DataFrame, p_target:str, p_title:str = None, p_comment:str = None):\n    print('----------------------------- show_pair_plot_vs_target -----------------------------')\n    # Create the grid\n    fig = plt.figure(figsize = (12, 12))\n    gs = fig.add_gridspec(1, 1)\n    background_color = \"#fbfbfb\"\n    # Prepare the grid\n    fig_desc = dict()\n    fig_desc['ax0'] = fig.add_subplot(gs[0, 0])\n    fig_desc['ax0'].set_facecolor(background_color)\n    fig_desc['ax0'].tick_params(axis = 'y', left=False)\n    fig_desc['ax0'].get_yaxis().set_visible(False)\n    for s in [\"top\", \"right\", \"left\"]:\n        fig_desc['ax0'].spines[s].set_visible(False)\n    # Draw plots\n    sns.pairplot(p_df, hue = p_target)\n    # Finalyze the figure\n    # Add Titles & Comments\n    if not p_title is None:\n        fig_desc['ax0'].text(-0.2, 0.4, p_title, fontsize = 20, fontweight = 'bold', fontfamily = 'serif')\n    if not p_comment is None:\n        fig_desc['ax0'].text(-0.2, 0.3, p_comment, fontsize = 13, fontweight = 'light', fontfamily = 'serif')   \n    plt.show()\n    # End of function show_pair_plot_vs_target\n\ndef show_mutual_information(p_df: pd.core.frame.DataFrame, p_title:str = None, p_comment:str = None, p_seed: int = SEED_HARCODED_VALUE) -> None:\n    print('----------------------------- show_mutual_information -----------------------------')\n    X = p_df.copy() # Do not modify input parameter\n    y = X.pop(TARGET_COLUMNS)\n    for c in X.select_dtypes(\"object\"): # Label encoding for categoricals\n        X[c], _ = X[c].factorize()\n    # Now all categorical features were transformed into numerical features, so discrete_features = True\n    if OUTPUT_IS_REGRESSION:\n        mi_scores = mutual_info_regression(X, y, discrete_features = True, random_state = p_seed)\n    else:\n        mi_scores = mutual_info_classif(X, y, discrete_features = True, random_state = p_seed)\n    mi_scores = pd.Series(mi_scores, name = 'MI Scores', index = X.columns)\n    mi_scores = mi_scores.sort_values(ascending = False)\n    print('show_mutual_information: MI Scores: ')\n    print(mi_scores)\n    # Create the grid\n    fig = plt.figure(figsize = (8, 8))\n    gs = fig.add_gridspec(1, 1)\n    background_color = \"#fbfbfb\"\n    # Prepare the grid\n    fig_desc = dict()\n    fig_desc['ax0'] = fig.add_subplot(gs[0, 0])\n    fig_desc['ax0'].set_facecolor(background_color)\n    #fig_desc['ax0'].tick_params(axis = 'y', left=False)\n    #fig_desc['ax0'].get_yaxis().set_visible(False)\n    for s in [\"top\", \"right\", \"left\"]:\n        fig_desc['ax0'].spines[s].set_visible(False)\n    # Draw plots\n    sns.barplot(mi_scores.values, mi_scores.index, orient = 'h')\n    # Finalyze the figure\n    # Add Titles & Comments\n    if not p_title is None:\n        fig_desc['ax0'].text(-0.2, 0.4, p_title, fontsize = 20, fontweight = 'bold', fontfamily = 'serif')\n    if not p_comment is None:\n        fig_desc['ax0'].text(-0.2, 0.3, p_comment, fontsize = 13, fontweight = 'light', fontfamily = 'serif')   \n    plt.show()\n    # End of function show_mutual_information\n\ndef cross_dataset_visualization(p_dfs: list, p_title:str = None, p_features:list = None, p_comment:str = None) -> None:\n    \"\"\"\n    This method plots compared features distribution according to the list (e.g. Train and Validation features)\n    \"\"\"\n    print('----------------------------- cross_dataset_visualization -----------------------------')\n    if p_features is None:\n        features = p_dfs[0].columns\n    else:\n        features = p_features\n    # Create the grid\n    fig, gs, fig_desc = create_grid(p_dfs[0][features])\n    # Draw plots\n    run_no = 0\n    for feature in features:\n        sns.histplot(p_dfs[0][feature].value_counts(), ax = fig_desc['ax' + str(run_no)], color='#ffd100')\n        sns.histplot(p_dfs[1][feature].value_counts(), ax = fig_desc['ax' + str(run_no)], color='#ff0100')\n        run_no += 1\n    # End of 'for' statement\n    # Finalyze the figure\n    finalize_grid((fig, gs, fig_desc), p_dfs[0], features, p_title, p_comment)\n    # End of function cross_dataset_visualization","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The kaggle_visualization() function provides different plot to explore the data distrubution (gaussian, exponecial...) and to detect outlier values. It will help 1) during the data cleaning and 2) later, to choose the ML algorithms (e.g. Outliers do not affect a tree-based algorithm).\nThere are two kind of data visualition:\n- The Univariate Plots which are related to each features, and\n- The Multivariate Plots which are related to interaction between features\n\nThe Univariate Plots:\n- Histograms: It provides a graphical representation of the distribution of a dataset. For a continuous numerical, it show the underlying frequency distribution or the probability  distribution of signal (see https://towardsdatascience.com/histograms-why-how-431a5cfbfcd5)\n- Density: It is the continuous form of the histogram (see above) and it shows an estimate of the continuous distribution of a feature (Gaussian distribution, exponential distribution...)\n\nThe Multivariate Plots\n- Correlationan: It provides indications about the changes between two features\n- scatter_matrix: It shows how much one feature is affected by another or the relationship between them","metadata":{}},{"cell_type":"code","source":"# b.1) Data visualizations\ndef kaggle_visualization(p_df: pd.core.frame.DataFrame, p_seed: int = SEED_HARCODED_VALUE) -> None:\n    \"\"\"\n    This method provides different views of the dataset (plot)\n    :parameters p_df: The dataset handle\n    :parameter p_seed: The seed value for reproductibility (TODO Not used)\n    \"\"\"\n    print('----------------------------- kaggle_visualization_data -----------------------------')\n    features = list(set(p_df.columns) - set(TARGET_COLUMNS))\n    categorical_columns = [col for col in p_df.columns if p_df[col].dtype == 'object']\n    if not DATE_TIME_COLUMNS is None:\n        categorical_columns = list(set(categorical_columns) - set(DATE_TIME_COLUMNS))\n    numerical_columns = [col for col in p_df.columns if p_df[col].dtype == 'int64' or p_df[col].dtype == 'float64']\n    print('kaggle_visualization: Features Distribution plots')\n    show_counts(p_df, p_hue = None)\n    # Histogram plots\n    print('kaggle_visualization: Numerical features Distribution plots')\n    show_distributions(p_df, p_features = numerical_columns)\n    if len(categorical_columns) != 0:\n        print('kaggle_visualization: Categorical features scatter plots')\n        show_scatter_plots(p_df, p_features = categorical_columns)\n    else:\n        print('kaggle_visualization: No categorical features to plots')\n    print('kaggle_visualization: Features outliers plots')\n    show_outliers(p_df, p_title = 'Features Distribution')\n    #show_trends(p_df, p_title = 'Features Distribution', p_comment = 'All features have bimodal or multimodal distribution')\n    print('kaggle_visualization: Histogram of each attributes regarding targets')\n    show_correlations(p_df)\n    print('kaggle_visualization: Features VS target distribution plots')\n    show_features_vs_target(p_df, p_target = TARGET_COLUMNS, p_features = numerical_columns)    \n    print('kaggle_visualization: Features VS target distribution plots')\n    show_pair_plot_vs_target(p_df, p_target = TARGET_COLUMNS)\n    print('kaggle_visualization: Done')\n    # End of function kaggle_visualization","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In case of images, we need to show them to understand the data and the problem to solve (e.g. flowers classification...).\nThis the role of the functions below.","metadata":{}},{"cell_type":"code","source":"# b.2) Images visualization\ndef kaggle_image_visualization(p_dataset, p_num_images: int = 10, p_prediction = None):\n    \"\"\"\n    This method show a number of images from p_dataset\n    :parameter p_dataset: The dataset containing the images to show. Note the dataset is usually already 'batch'\n    :parameter p_num_images: The number of images to show\n    :parameter p_prediction: \n    \"\"\"\n    print('----------------------------- kaggle_image_visualization -----------------------------')\n    # Peek some data from the dataset\n    it = iter(p_dataset.unbatch().batch(p_num_images))\n    batch = next(it)\n    kaggle_display_batch(batch, p_prediction)\n    # End of function kaggle_image_visualization\n\ndef kaggle_batch_to_numpy(p_dataset):\n    print('----------------------------- kaggle_batch_to_numpy -----------------------------')\n    images, labels = p_dataset\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case,\n                                     # these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is\n    # the case for test data)\n    return numpy_images, numpy_labels\n    # End of function kaggle_batch_to_numpy\n\ndef kaggle_title_from_label(p_label: str, p_correct_label: str):\n    print('----------------------------- kaggle_title_from_label -----------------------------')\n    if p_correct_label is None:\n        return IMAGE_CLASSES[p_label], True\n    correct = (p_label == p_correct_label)\n    return \"{} [{}{}{}]\".format(IMAGE_CLASSES[p_label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                IMAGE_CLASSES[p_correct_label] if not correct else ''), correct\n    # End of function kaggle_title_from_label\n\ndef kaggle_display_batch_image(p_image, p_title:str, subplot, red = False, titlesize = 16, p_cmap = 'viridis'):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(p_image, cmap = p_cmap)\n    if len(p_title) > 0:\n        plt.title(p_title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    # End of function kaggle_display_batch_image\n    \ndef kaggle_display_batch(p_databatch, p_predictions = None):\n    \"\"\"\n    \"\"\"\n    print('----------------------------- kaggle_display_batch -----------------------------')\n    # Images\n    images, labels = kaggle_batch_to_numpy(p_databatch)\n    if labels is None: # Fill with None\n        labels = [None for _ in enumerate(images)] \n    # Auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n    # Sizing and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows, cols, 1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE / cols * rows))\n    else:\n        plt.figure(figsize=(FIGSIZE / rows * cols,FIGSIZE))\n    # Display\n    for i, (image, label) in enumerate(zip(images[:rows * cols], labels[:rows * cols])):\n        title = 'No label' if label is None or type(label) != 'int' else IMAGE_CLASSES[label]\n        correct = True\n        if not p_predictions is None:\n            title, correct = kaggle_title_from_label(p_predictions[i], label)\n        dynamic_titlesize = FIGSIZE * SPACING / max(rows, cols) * 40 + 3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = kaggle_display_batch_image(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    # Layout\n    plt.tight_layout()\n    if label is None and p_predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n    # End of function kaggle_display_batch\n\ndef kaggle_display_image_and_component(p_image, p_title:str = None) -> None:\n    \"\"\"\n    This function displays an image and its RGB components separatly\n    :parameter p_image: The image to display (RGB format)\n    :parameter p_title: The title of the display\n    \"\"\"\n    # Extract RGB components\n    pixels = img_to_array(p_image)\n    red = pixels[:, :, 0]\n    green = pixels[:, :, 1]\n    blue = pixels[:, :, 2]\n    # Sizing and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    rows = 1\n    cols = 4\n    subplot=(rows, cols, 1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE / cols * rows))\n    else:\n        plt.figure(figsize=(FIGSIZE / rows * cols,FIGSIZE))\n    # Display image and its components\n    images = (pixels, red, green, blue)\n    titles = (p_title, 'red component', 'green component', 'blue component')\n    for i, (image, title) in enumerate(zip(images, titles)):\n        dynamic_titlesize = FIGSIZE * SPACING / max(rows, cols) * 40 + 3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = kaggle_display_batch_image(image, title, subplot, True, titlesize = dynamic_titlesize, p_cmap = 'viridis' if i == 0 else 'gray')    \n    # Layout\n    plt.tight_layout()\n    plt.subplots_adjust(wspace=0, hspace=0)\n    plt.show()\n    # End of function kaggle_display_image_and_component\n\ndef kaggle_display_confusion_matrix(cmat, score, precision, recall) -> None:\n    \"\"\"\n    \"\"\"\n    print('----------------------------- kaggle_display_confusion_matrix -----------------------------')\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(IMAGE_CLASSES)))\n    ax.set_xticklabels(IMAGE_CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(IMAGE_CLASSES)))\n    ax.set_yticklabels(IMAGE_CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()\n    print('kaggle_display_confusion_matrix: Done')\n    # End of function kaggle_display_confusion_matrix\n\ndef kaggle_display_learning_curves(training, validation, title, subplot) -> None:\n    print('----------------------------- kaggle_display_learning_curves -----------------------------')\n    if subplot % 10 == 1: # set up the subplots on the first call\n        plt.subplots(figsize = (10,10), facecolor = '#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])\n    plt.show()\n    print('kaggle_display_learning_curves: Done')\n    # End of function kaggle_display_learning_curves","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we need to do a break and we need:\n1. to understand exactly what each column is?\n2. to learn from the results we got\n\n\n# Iris classification\n\nTODO\n\n# Titanic Desaster\n\nTODO\n\n# PIMA diabetes\nLet's take a look to the number of '0' value in each columns (see kaggle_summurize_data/Zeros per columns distribution and features distributions) and try to understand what does it mean? \n- What does it mean a Glucose (plass) or a Blood pressure (pres) value of 0? It's not possible!!!\n- Idem for Skin thickness\n\nIn this case, one solution is to replace '0' by a NaN value and the Impute process will do the job ;)\n\nThis is done in kaggle_pre_features_engineering() function where 0 values are replaced by NaN values for the following features: 'plas', 'pres', 'skin', 'test' and 'mass'\n\n# Flower classification using TPU\nHere the problem is to classify a photo of a flower in the right category (see IMAGE_CLASSES).\nThe datasets (training and validation) are in Tensorflow tfrec format (see labeled_image_feature_description) and the test dataset (having the role of unseen data) is also in tfrec format (see unlabeled_image_feature_description).\n\n\n*TODO: Add data analyze results for other dataset (Iris, Melbourne Housing Prices)*\n","metadata":{}},{"cell_type":"markdown","source":"The function kaggle_ml_quick_and_dirty() provides a 'quick and dirty' evaluation of a ML based on RandomForestClassifier algorithm with estimators parameter set to 128. All rows with NaN values are removed and all categorical attributes are excluded.\n\nThe idea is to use this function to establish a baseline by training a basic model at different steps of the Feature engineering (such as un-augmented dataset, data cleanup...). A baseline score can help us to decide whether the different processings are actually useful or worth.","metadata":{}},{"cell_type":"code","source":"# c.1) Basic ML for a quick & dirty evaluation\ndef kaggle_ml_quick_and_dirty(p_train_df: pd.core.frame.DataFrame, \n                              p_validation_df: pd.core.frame.DataFrame, \n                              p_test_df: pd.core.frame.DataFrame = None,\n                              p_test_outputs_df: pd.core.frame.DataFrame = None,\n                              p_seed:int = SEED_HARCODED_VALUE\n                             ) -> np.ndarray:\n    \"\"\"\n    This method provides a first ML evalulation based on RandomForest algorithm\n    :parameters p_train_df: The Training dataset (to fit the model)\n    :parameters p_validation_df: The Training dataset (to validate the model)\n    :parameters p_test_df: The Test dataset (to do prediction on unseen data). Default: None\n    :parameters p_test_outputs_df: The outputs for the Test dataset. It will be None in case of Kaggle compete. Default: None\n    :parameter p_seed: The seed value for reproductibility\n    :return: The machine learning model  \n    \"\"\"\n    print('----------------------------- kaggle_ml_quick_and_dirty -----------------------------')\n\n    l = LabelEncoder()\n    # Build training & validation datasets\n    p = p_train_df.copy()\n    # Remove NaN values\n    p.dropna(axis = 0, inplace = True)\n    # Ignore categorical values\n    if p[TARGET_COLUMNS].dtype == 'object':\n        # Do basic imputation\n        p[TARGET_COLUMNS] = l.fit_transform(p[TARGET_COLUMNS])\n    p = p.select_dtypes(exclude=['object'])\n    Y_train = p[TARGET_COLUMNS]\n    X_train = p.drop([TARGET_COLUMNS], axis = 1)\n    print('X_train =>', X_train.head())\n    print('Y_train =>', Y_train.head())\n    print('type(Y_train) =>', type(Y_train))\n\n    p = p_validation_df.copy()\n    # Remove NaN values\n    p.dropna(axis = 0, inplace = True)\n    # Ignore categorical values\n    if p[TARGET_COLUMNS].dtype == 'object':\n        # Do basic imputation\n        p[TARGET_COLUMNS] = l.fit_transform(p[TARGET_COLUMNS])\n    p = p.select_dtypes(exclude=['object'])\n    Y_validation = p[TARGET_COLUMNS]\n    X_validation = p.drop([TARGET_COLUMNS], axis = 1)\n    print('X_validation =>', X_validation.head())\n    print('Y_validation =>', Y_validation.head())\n    print('type(Y_validation) =>', type(Y_validation))\n\n    # Use classical model\n    model = None\n    if OUTPUT_IS_REGRESSION:\n        model = ensemble.RandomForestRegressor(n_estimators = 128, max_depth = 16, max_features = 4, random_state = p_seed)\n    else:\n        model = ensemble.RandomForestClassifier(n_estimators = 128, max_depth = 16, max_features = 4, random_state = p_seed)\n    # Train the model\n    model.fit(X_train, Y_train.ravel())\n    # Do predictions\n    y_predictions = model.predict(X_validation)\n    # Get scoring\n    get_scoring(Y_validation, y_predictions)\n    \n    # Do prediction with unseen data\n    if not p_test_df is None:\n        p = p_test_df.copy()\n        # Remove NaN values\n        p.dropna(axis = 0, inplace = True) # Buggy, because #rows X_test will be different than #rows Y_test, try Impute or drop same rows in both X_test & Y_test\n        # Ignore categorical values\n        p = p.select_dtypes(exclude=['object'])\n        y_predictions = model.predict(p)\n        # Evaluate the results when it is possible\n        if not p_test_outputs_df is None:\n            print('kaggle_ml_quick_and_dirty: p shape: ', p.shape)\n            print('kaggle_ml_quick_and_dirty: p_test_outputs_df: %s' % str(p_test_outputs_df.shape))\n            print('kaggle_ml_quick_and_dirty: y_predictions: %s' % str(y_predictions.shape))\n            # Get scoring\n            get_scoring(p_test_outputs_df, y_predictions)\n\n    print('kaggle_ml_quick_and_dirty: Done')\n    return model\n    # End of function kaggle_ml_quick_and_dirty\n\ndef get_scoring(p_expected_validation: np.array, p_predictions: np.array) -> None:\n    \"\"\"\n    Display scores of the prediction\n    :parameter p_expected_validation: The expected predictions\n    :parameter p_predictions: The ML predictions\n    \"\"\"\n    if OUTPUT_IS_REGRESSION:\n        print('get_scoring: Model R2 score=%f' % (r2_score(p_expected_validation, p_predictions)))\n        print('get_scoring: Model Mean absolute error regression loss (MAE): %0.4f' % mean_absolute_error(p_expected_validation, p_predictions))\n        print('get_scoring: Model Mean squared error regression loss (MSE): %0.4f' % mean_squared_error(p_expected_validation, p_predictions))\n        print('get_scoring: Mean squared error regression loss (RMSE): %0.4f' % np.sqrt(mean_squared_error(p_expected_validation, p_predictions)))\n    else:\n        print('get_scoring: Model accuracy score: %0.4f' % accuracy_score(p_expected_validation, p_predictions))\n        print('get_scoring: ROC=%s' %(roc_auc_score(p_expected_validation, p_predictions)))\n        print('get_scoring: Model F1 score=%f' % (f1_score(p_expected_validation, p_predictions)))\n        print('get_scoring: Confusion matrix: %s' % str(confusion_matrix(p_expected_validation, p_predictions)))\n        print('get_scoring: Classification report:\\n%s' % str(classification_report(p_expected_validation, p_predictions)))\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The set of functions below is specific to Neural Network learning. It provides callbacks to create DL models and functions to visualize learning curves or learning rate curves (see point c.2 below).","metadata":{}},{"cell_type":"code","source":"# Helper functions for kaggle_dl_quick_and_dirty()\ndef kaggle_build_pretrained_model(\n                                  p_pretrained_layers:list, \n                                  p_image_size: list,\n                                  p_channels: int = 3,\n                                  p_augmentations: list = None,\n                                  p_seed: int = SEED_HARCODED_VALUE\n                                  ) -> tf.keras.Sequential:\n    \"\"\"\n    This function builds the pretrained models stacks for the Neural Network\n    :parameter p_pretrained_layers: The pretrained models to be used\n    :parameter p_image_size: The image size\n    :parameter p_channels: The number of channels (e.g. 1 for grayscale, 3 for RGB)\n    :return: The pretrained models stacks to build the final Neural Network\n    \"\"\"\n    \n    # Create model with Resizing and rescaling layers (always active)\n    # See https://www.tensorflow.org/tutorials/images/data_augmentation\n    model = tf.keras.Sequential();\n    resize_and_rescale = tf.keras.Sequential([\n        tf.keras.layers.experimental.preprocessing.Resizing(*IMAGE_SIZE),\n        tf.keras.layers.experimental.preprocessing.Rescaling(1./255.0)\n    ])\n    model.add(resize_and_rescale)\n    if not p_augmentations is None:\n        data_augmentation = tf.keras.Sequential()\n        for augmentation in p_augmentations:\n            if augmentation == 'flip_right_left':\n                data_augmentation.add(tf.keras.layers.experimental.preprocessing.RandomFlip('horizontal', p_seed))\n            elif augmentation == 'flip_up_down':\n                data_augmentation.add(tf.keras.layers.experimental.preprocessing.RandomFlip('vertical', p_seed))\n            elif augmentation == 'rotate':\n                data_augmentation.add(tf.keras.layers.experimental.preprocessing.RandomRotation(0.2, p_seed))\n            elif augmentation == 'random_contrast':\n                data_augmentation.add(tf.keras.layers.experimental.preprocessing.RandomContrast(0.2, p_seed))\n            elif augmentation == 'random_brightness':\n                pass\n            else:\n                raise Exception('kaggle_build_pretrained_model', 'Wrong augmentation parameter')\n            # End of 'for' statement\n        model.add(data_augmentation)\n    else:\n        pass #raise Exception('kaggle_build_pretrained_model', 'Wrong parameters')\n    for p in p_pretrained_layers:\n        if p == 'DenseNet201': \n            pretrained_model = tf.keras.applications.DenseNet201(\n                weights = 'imagenet',\n                include_top = False,\n                pooling = 'max',\n                input_shape = [*p_image_size, p_channels]\n            )\n            pretrained_model.trainable = False\n            model.add(pretrained_model)\n            #model.add(tf.keras.layers.GlobalAveragePooling2D())\n        elif p == 'InceptionV3': \n            # See https://github.com/fchollet/deep-learning-models/blob/ccd0eb24996b4cbff4231b90cd44b057c0b20f14/inception_v3.py\n            pretrained_model = tf.keras.applications.InceptionV3(\n                weights = 'imagenet',\n                include_top = False,\n                pooling = 'max',\n                input_shape = [*p_image_size, p_channels]\n            )\n            pretrained_model.trainable = False\n            model.add(pretrained_model)\n        elif p == 'MobileNetV2':\n            pretrained_model = tf.keras.applications.MobileNetV2(\n                weights = 'imagenet',\n                include_top = False,\n                pooling = 'max',\n                input_shape = [*p_image_size, p_channels]\n            )\n            pretrained_model.trainable = False\n            model.add(pretrained_model)\n        elif p == 'ResNet50':\n            # See https://github.com/fchollet/deep-learning-models/blob/ccd0eb24996b4cbff4231b90cd44b057c0b20f14/resnet50.py\n            pretrained_model = tf.keras.applications.ResNet50(\n                weights = 'imagenet',\n                include_top = False,\n                input_shape = [*p_image_size, p_channels]\n            )\n            pretrained_model.trainable = False\n            model.add(pretrained_model)\n        elif p == 'VGG16': \n            # See https://github.com/fchollet/deep-learning-models/blob/master/vgg16.py\n            pretrained_model = tf.keras.applications.VGG16(\n                weights = 'imagenet',\n                include_top = False,\n                input_shape = [*p_image_size, p_channels]\n            )\n            pretrained_model.trainable = False\n            model.add(pretrained_model)\n        elif p == 'VGG19':\n            # See https://github.com/fchollet/deep-learning-models/blob/master/vgg19.py\n            pretrained_model = tf.keras.applications.VGG19(\n                weights = 'imagenet',\n                include_top = False,\n                pooling = 'max',\n                input_shape = [*p_image_size, p_channels]\n            )\n            pretrained_model.trainable = False\n            model.add(pretrained_model)\n        else:\n            raise Exception('kaggle_build_pretrained_model', 'Undefined pretrained model')\n        # End of 'for' statement\n    return model\n    # End of function kaggle_build_pretrained_model\n\ndef kaggle_create_sequential_classifier_model(\n                                              p_optimizer: str = 'adam', \n                                              p_loss: str = 'binary_crossentropy', \n                                              p_metrics: list = ['AUC', 'accuracy'],\n                                              p_pretrained_layers:list = None,\n                                              p_image_size: list = None,\n                                              p_channels: int = 3,\n                                              p_class_num: int = 1,\n                                              p_augmentations: list = None\n                                             ) -> tf.keras.Sequential:\n    \"\"\"\n    Build a Neural network model for classification\n    \"\"\"\n    print('----------------------------- kaggle_create_sequential_classifier_model -----------------------------')\n    global DL_INPUT_SHAPE, DL_DROP_RATE\n\n    model = None\n    if p_pretrained_layers is None:\n        model = tf.keras.Sequential([\n                                    tf.keras.layers.BatchNormalization(input_shape = DL_INPUT_SHAPE),\n                                    tf.keras.layers.Dense(64, activation = 'relu'),\n                                    tf.keras.layers.Dropout(rate = DL_DROP_RATE),\n                                    tf.keras.layers.BatchNormalization(),\n                                    tf.keras.layers.Dense(64, activation = 'relu'),\n                                    tf.keras.layers.Dropout(rate = DL_DROP_RATE),\n                                    tf.keras.layers.BatchNormalization(),\n                                    tf.keras.layers.Dense(64, activation = 'relu'),\n                                    tf.keras.layers.BatchNormalization(),\n                                    tf.keras.layers.Dropout(rate = DL_DROP_RATE),\n                                    tf.keras.layers.Dense(p_class_num, activation = 'sigmoid' if p_class_num == 1 else 'softmax'), # Binary classes or Multi classes\n                                    ])\n    else:\n        model = kaggle_build_pretrained_model(p_pretrained_layers, p_image_size = p_image_size, p_augmentations = p_augmentations)\n        model.add(tf.keras.layers.BatchNormalization())\n        model.add(tf.keras.layers.Dropout(rate = DL_DROP_RATE)),\n        model.add(tf.keras.layers.Dense(64, activation='relu', kernel_regularizer = tf.keras.regularizers.l1_l2(0.01)))\n        model.add(tf.keras.layers.BatchNormalization())\n        model.add(tf.keras.layers.Dropout(rate = DL_DROP_RATE)),\n        model.add(tf.keras.layers.Dense(64, activation='relu', kernel_regularizer = tf.keras.regularizers.l1_l2(0.01)))\n        model.add(tf.keras.layers.BatchNormalization())\n        model.add(tf.keras.layers.Dropout(rate = DL_DROP_RATE)),\n        model.add(tf.keras.layers.Dense(64, activation='relu', kernel_regularizer = tf.keras.regularizers.l1_l2(0.01)))\n        model.add(tf.keras.layers.BatchNormalization())\n        model.add(tf.keras.layers.Dropout(rate = DL_DROP_RATE)),\n        model.add(tf.keras.layers.Dense(64, activation='relu', kernel_regularizer = tf.keras.regularizers.l1_l2(0.01)))\n        model.add(tf.keras.layers.BatchNormalization())\n        model.add(tf.keras.layers.Dropout(rate = DL_DROP_RATE)),\n        model.add(tf.keras.layers.Dense(p_class_num, activation = 'sigmoid' if p_class_num == 1 else 'softmax')) # Binary classes or Multi classes\n        model.build(input_shape = [None, *p_image_size, p_channels])\n\n    model.compile(optimizer = p_optimizer, loss = p_loss, metrics = p_metrics)\n\n    print('kaggle_create_sequential_classifier_model: Model summary:')\n    model.summary()\n    #plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)\n    return model\n    # End of function kaggle_create_sequential_classifier_model\n\ndef kaggle_create_sequential_regressor_model(\n                                             p_optimizer: str = 'adam', \n                                             p_loss: str = 'mae', \n                                             p_metrics: list = ['mae'],\n                                             p_pretrained_layers: list = None,\n                                             p_image_size: list = None,\n                                             p_channels: int = 3,\n                                             p_class_num: int = 1,\n                                             p_augmentations: list = None\n                                            ) -> tf.keras.Sequential:\n    \"\"\"\n    Build a Neural network model for regression\n    \"\"\"\n    print('----------------------------- kaggle_create_sequential_regressor_model -----------------------------')\n    global DL_INPUT_SHAPE, DL_DROP_RATE\n\n    model = None\n    if p_pretrained_layers is None:\n        model = tf.keras.Sequential([\n                                    tf.keras.layers.BatchNormalization(input_shape = DL_INPUT_SHAPE),\n                                    tf.keras.layers.Dense(64, activation = 'relu'),\n                                    tf.keras.layers.Dropout(rate = DL_DROP_RATE),\n                                    tf.keras.layers.BatchNormalization(),\n                                    tf.keras.layers.Dense(64, activation = 'relu'),\n                                    tf.keras.layers.BatchNormalization(),\n                                    tf.keras.layers.Dropout(rate = DL_DROP_RATE),\n                                    tf.keras.layers.Dense(64, activation = 'relu'),\n                                    tf.keras.layers.BatchNormalization(),\n                                    tf.keras.layers.Dropout(rate = DL_DROP_RATE),\n                                    tf.keras.layers.Dense(p_class_num, activation = 'relu'),\n                                    ])\n    else:\n        model = kaggle_build_pretrained_model(p_pretrained_layers, p_image_size = p_image_size, p_augmentations = p_augmentations);\n        model.add(tf.keras.layers.BatchNormalization())\n        model.add(tf.keras.layers.Dropout(rate = DL_DROP_RATE)),\n        model.add(tf.keras.layers.Dense(64, activation='relu', kernel_regularizer = tf.keras.regularizers.l1_l2(0.01)))\n        model.add(tf.keras.layers.BatchNormalization())\n        model.add(tf.keras.layers.Dropout(rate = DL_DROP_RATE)),\n        model.add(tf.keras.layers.Dense(64, activation='relu', kernel_regularizer = tf.keras.regularizers.l1_l2(0.01)))\n        model.add(tf.keras.layers.BatchNormalization())\n        model.add(tf.keras.layers.Dropout(rate = DL_DROP_RATE)),\n        model.add(tf.keras.layers.Dense(64, activation='relu', kernel_regularizer = tf.keras.regularizers.l1_l2(0.01)))\n        model.add(tf.keras.layers.BatchNormalization())\n        model.add(tf.keras.layers.Dropout(rate = DL_DROP_RATE)),\n        model.add(tf.keras.layers.Dense(64, activation='relu', kernel_regularizer = tf.keras.regularizers.l1_l2(0.01)))\n        model.add(tf.keras.layers.BatchNormalization())\n        model.add(tf.keras.layers.Dropout(rate = DL_DROP_RATE)),\n        model.add(tf.keras.layers.Dense(p_class_num, activation='relu'))\n        model.build(input_shape = [None, *p_image_size, p_channels])\n\n    model.compile(optimizer = p_optimizer, loss = p_loss, metrics = p_metrics)\n\n    print('kaggle_create_sequential_regressor_model: Model summary:')\n    model.summary()\n    return model\n    # End of function kaggle_create_sequential_regressor_model\n\nLR_ALG = 'rop'\ndef kaggle_build_learning_rate_schedule(epoch: int,\n                                        start_lr = 0.0005, \n                                        min_lr: float = 0.0001, \n                                        max_lr = 0.001,\n                                        rampup_epochs: int = 5, \n                                        sustain_epochs: int = 0,\n                                        exp_decay: float = 0.8,\n                                        patience: int = 1,\n                                        factor: float = 0.5\n                                       ):\n    \"\"\"\n    \"\"\"\n    global LR_ALG\n    def _exponential_lr(epoch, start_lr, min_lr, max_lr, rampup_epochs, sustain_epochs, exp_decay):\n        \"\"\"\n        \"\"\"\n        # linear increase from start to rampup_epochs\n        print('kaggle_build_learning_rate_schedule._exponential_lr: epoch=%d' % epoch)\n        if epoch < rampup_epochs:\n            lr = ((max_lr - start_lr) /\n                  rampup_epochs * epoch + start_lr)\n        # constant max_lr during sustain_epochs\n        elif epoch < rampup_epochs + sustain_epochs:\n            lr = max_lr\n        # exponential decay towards min_lr\n        else:\n            lr = ((max_lr - min_lr) *\n                  exp_decay**(epoch - rampup_epochs - sustain_epochs) +\n                  min_lr)\n        return lr\n\n    #print('kaggle_build_learning_rate_schedule: alg=%s' % LR_ALG)\n    if LR_ALG == 'exp': # Exponential LR curve\n        l = _exponential_lr(epoch, start_lr, min_lr, max_lr, rampup_epochs, sustain_epochs, exp_decay)\n    else:\n        raise Exception('kaggle_build_learning_rate_schedule', 'Wrong parameters')\n    return l\n    # End of function kaggle_build_learning_rate_schedule\n\ndef kaggle_display_learning_rate_curve(p_lr):\n    print('----------------------------- kaggle_display_learning_rate_curve -----------------------------')\n    rng = [i for i in range(DL_EPOCH_NUM)]\n    #print('rng ==> ', rng)\n    y = [p_lr(x) for x in rng]\n    #print('y ==> ', y)\n    plt.plot(rng, y)\n    plt.show()\n    print('kaggle_display_learning_rate_curve: Done: Schedule: {:.3g} to {:.3g} to {:.3g}'.format(y[0], max(y), y[-1]))\n    # End of function kaggle_display_learning_rate_curve","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The function kaggle_dl_quick_and_dirty() provides a 'quick and dirty' evaluation of a DL based on a basic Neural Network. All rows with NaN values are removed and all categorical attributes are excluded.","metadata":{}},{"cell_type":"code","source":"# c.2) Basic DL for a quick & dirty evaluation\ndef kaggle_dl_quick_and_dirty(p_train_df: pd.core.frame.DataFrame, \n                              p_validation_df: pd.core.frame.DataFrame, \n                              p_strategy,\n                              p_test_df: pd.core.frame.DataFrame = None,\n                              p_test_outputs_df: pd.core.frame.DataFrame = None,\n                              p_seed:int = SEED_HARCODED_VALUE,\n                             ) -> np.ndarray:\n    \"\"\"\n    This method provides a first DL evalulation based on RandomForest algorithm\n    :parameters p_train_df: The Training dataset (to fit the model)\n    :parameters p_validation_df: The Training dataset (to validate the model)\n    :parameters p_train_df: The Test dataset (to do prediction). Default: None\n    :parameters p_test_outputs_df: The outputs for the Test dataset. It will be None in case of Kaggle compete. Default: None\n    :parameters p_strategy: . Default: None\n    :parameter p_seed: The seed value for reproductibility\n    :return: The machine learning model  \n    \"\"\"\n\n    print('----------------------------- kaggle_dl_quick_and_dirty -----------------------------')\n    global DL_INPUT_SHAPE, DL_DROP_RATE\n\n    l = LabelEncoder()\n    # Build training & validation datasets\n    p = p_train_df.copy()\n    # Remove NaN values\n    p.dropna(axis = 0, inplace = True)\n    # Ignore categorical values\n    if p[TARGET_COLUMNS].dtype == 'object':\n        # Do basic imputation\n        p[TARGET_COLUMNS] = l.fit_transform(p[TARGET_COLUMNS])\n    p = p.select_dtypes(exclude=['object'])\n    Y_train = p[TARGET_COLUMNS]\n    X_train = p.drop([TARGET_COLUMNS], axis = 1)\n\n    p = p_validation_df.copy()\n    # Remove NaN values\n    p.dropna(axis = 0, inplace = True)\n    # Ignore categorical values\n    if p[TARGET_COLUMNS].dtype == 'object':\n        # Do basic imputation\n        p[TARGET_COLUMNS] = l.fit_transform(p[TARGET_COLUMNS])\n    p = p.select_dtypes(exclude=['object'])\n    Y_validation = p[TARGET_COLUMNS]\n    X_validation = p.drop([TARGET_COLUMNS], axis = 1)\n\n    # Use classical model\n    #   Use global valriables instead of parameters due to folling SKlearn issue\n    #   KerasClassifier with build_fn a class does not work with sklearn.model_selection.cross_val_score #13717\n    #   See https://github.com/keras-team/keras/issues/13717\n    DL_INPUT_SHAPE = [X_train.shape[1]]\n    with p_strategy.scope():\n        if OUTPUT_IS_REGRESSION:\n            model = KerasRegressor(build_fn = kaggle_create_sequential_regressor_model)#p_strategy, p_input_shape = [X_train.shape[1]])\n        else:\n            model = KerasClassifier(build_fn = kaggle_create_sequential_classifier_model)#p_strategy, p_input_shape = [X_train.shape[1]])\n    with p_strategy.scope():\n        # Train the model\n        early_stopping = EarlyStopping(\n            min_delta = 0.001, # minimium amount of change to count as an improvement\n            patience = 20, # how many epochs to wait before stopping\n            restore_best_weights = True,\n        )\n        history = model.fit(X_train, \n                            Y_train, \n                            validation_data=(X_validation, Y_validation), \n                            epochs = DL_EPOCH_NUM, \n                            batch_size = DL_BATCH_SIZE * p_strategy.num_replicas_in_sync, \n                            callbacks = [early_stopping],\n                            verbose = 2\n                           )\n    # Draw Loss and AUC\n    print('kaggle_dl_quick_and_dirty: Draw Loss, Accuracy and AUC')\n    history_df = pd.DataFrame(history.history)\n    print(history_df.head())\n    print(\"kaggle_dl_quick_and_dirty: Minimum validation loss: {}\".format(history_df['val_loss'].min()))\n    kaggle_display_learning_curves(\n        history.history['loss'],\n        history.history['val_loss'],\n        'loss',\n        211,\n    )\n    print(\"kaggle_dl_quick_and_dirty: Maximum validation Accuracy: {}\".format(history_df['val_accuracy'].max()))\n    kaggle_display_learning_curves(\n        history.history['auc'],\n        history.history['val_auc'],\n        'accuracy',\n        212,\n    )\n    print(\"kaggle_dl_quick_and_dirty: Maximum validation AUC: {}\".format(history_df['val_auc'].max()))\n    # Do predictions\n    print('kaggle_dl_quick_and_dirty: Do predictions')\n    y_predictions = model.predict(X_validation)\n    # Get scoring\n    get_scoring(Y_validation, y_predictions)\n\n    # Do prediction with unseen data\n    if not p_test_df is None:\n        p = p_test_df.copy()\n        # Remove NaN values\n        p.dropna(axis = 0, inplace = True)\n        # Ignore categorical values\n        p = p.select_dtypes(exclude=['object'])\n        y_predictions = model.predict(p)\n    # Evaluate the results when it is possible\n    if not p_test_outputs_df is None:\n        # Get scoring\n        get_scoring(p_test_outputs_df, y_predictions)\n\n    print('kaggle_dl_quick_and_dirty: Done')\n    return model\n    # End of function kaggle_dl_quick_and_dirty","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, the next step is to prepare the data for ML. Usually, you have better result when all the features (features and outputs) are in numerical format (int or float).\n\n1. Feature engineering. It eliminates NULL or NaN values, duplicate values, and it transforms date/time column, categorical columns into numerical fetures. It identifies & handles outliers... (3.a). Categorical columns are usually of type object and the objective here is to transform these categorical columns into numerical columns. Date/time columns can be either object (e.g. date/time in string format) of type datetime64[ns]. For sepcific features such as 'Age', it is possible to create new feature grouping the Age values per range, between from the lower Age value to the upper Age value\n2. Data transformation. It applies some numerical transformation such as standardization of features... (3.b)\n3. Features selection. It selects and prepares the dataset for the training and the validation (3.c)","metadata":{}},{"cell_type":"code","source":"# 3. Prepare the dataset for your Machine Learning processing\n# a) Data Cleaning\nEncoders = dict()\nEncoder_Instance = LabelEncoder() # Use global variable for future reverse features engineering\nImputer_Instance = None\ndef kaggle_features_engineering(p_df: pd.core.frame.DataFrame,                               \n                                p_missing_value_method: str = 'drop_columns', \n                                p_duplicated_value_method: str = 'drop_columns', \n                                p_categorical_onehot_threshold: int = 10, \n                                p_date_time_columns: list = None, \n                                p_date_time_engineering: str = 'python_time',\n                                p_outliers_lower_percentile = 25,\n                                p_outliers_upper_percentile = 75,\n                                p_outliers_impute_method = 'mean'\n                               ) -> pd.core.frame.DataFrame:\n    \"\"\"\n    This function performs a cleaning of the dataset to remove null values, duplicate values, based on the specified method\n    :parameters p_df: The dataset handle\n    :parameters p_missing_value_method: The method to cleanup NaN values in the dataset ('drop_columns', 'drop_lines', 'mean', 'median'). Default: 'drop_columns'\n    :parameters p_duplicated_value_method: The method to cleanup duplicated in the dataset ('drop_columns', 'drop_lines', 'mean', 'median'). Default: 'drop_columns'\n    :parameters p_categorical_onehot_threshold: The maximum cardinality to apply OneHotEncoder to a categorical variable. Defaut: 10\n    :parameters p_date_time_engineering: The method to convert Date/Time. Default: 'python_time'\n    :parameters p_outliers_lower_percentile: Percentile threshold for the Q1 (lower bound). Default: 25\n    :parameters p_outliers_upper_percentile: Percentile threshold for the Q3 (upper bound). Default: 75\n    :parameters p_outliers_impute_method: Method to use to impute outliers ('mean' or 'median'). Default: 'mean'\n    :return: The dataset after the cleanup process\n    \"\"\"\n    global Encoders, Encoder_Instance, Imputer_Instance\n    \n    print('----------------------------- kaggle_features_engineering -----------------------------')\n    # Cleanup dataset\n    old_shape = p_df.shape\n    p = p_df.copy() # The final dataset\n\n    # Apply feature processing resulting of the data analyzing\n    p = kaggle_pre_features_engineering(p)\n\n    # Build the list of categorical and numerical features\n    categorical_columns = [col for col in p.columns if p[col].dtype == 'object']\n    numerical_columns = [col for col in p.columns if p[col].dtype == 'int64' or p[col].dtype == 'float64']\n    print('kaggle_features_engineering: Initial categorical_columns:')\n    print(categorical_columns)\n    print('kaggle_features_engineering: Initial numerical_columns:')\n    print(numerical_columns)\n    if p_date_time_columns is not None:\n        if len(categorical_columns) != 0:\n            categorical_columns = list(set(categorical_columns) - set(p_date_time_columns))\n        numerical_columns = list(set(numerical_columns) - set(p_date_time_columns))\n\n    # Convert Date/time columns\n    # dtype = 'datetime64[ns]'\n    print('----------------------------- kaggle_features_engineering: Processing Date/Time columns')\n    if p_date_time_columns is not None: # Process specified columns\n        # Check date/time formats\n        for column in p_date_time_columns: # TODO Check if all DateTime values have the same format, i.e. same length\n            date_lengths = p[column].str.len().value_counts()\n            print('kaggle_features_engineering: %s lengths:' % column)\n            print('%s - %d' % (str(date_lengths), len(date_lengths)))\n        # End of 'for' statement\n        p[p_date_time_columns] = p[p_date_time_columns].astype('datetime64[ns]')\n        p[p_date_time_columns] = p[p_date_time_columns].astype('int64')\n        print('kaggle_features_engineering: Date/time columns processed')\n    else:\n        print('kaggle_features_engineering: No date/time values')\n    # Be sure there is no more 'datetime64[ns]' types in the dataset\n    datetime_columns = [col for col in p.columns if p[col].dtype == 'datetime64[ns]']\n    if len(datetime_columns) != 0:\n        raise Exception('kaggle_features_engineering: There still has datetime64[ns] type in dataset', 'method=%s' % str(p.info()))\n\n    # Find N/A values for categorical columns and replace them by the value with the higher frequency\n    print('----------------------------- kaggle_features_engineering: Processing NaN values')\n    categorical_columns_with_nan = [col for col in p.columns if p[col].dtype == 'object' and p[col].isna().sum() != 0]\n    if len(categorical_columns_with_nan) != 0:\n        print('----------------------------- kaggle_features_engineering: Impute NaN values for categorical columns with MAX value')\n        for col in categorical_columns_with_nan:\n            p[col].fillna(p[col].value_counts().idxmax(), inplace = True)\n        # End of 'for'statement\n        # Check that there are no more categorical columns with NaN\n        categorical_columns_with_nan = [col for col in p.columns if p[col].dtype == 'object' and p[col].isna().sum() != 0]\n        if len(categorical_columns_with_nan) != 0:\n            raise Exception('kaggle_features_engineering: There still has categorical columns with NaN value in dataset', 'method=%s' % str(categorical_columns_with_nan))\n    else:\n        print('----------------------------- kaggle_features_engineering: No NaN value in categorical columns')\n    # Use Imputation to replace NaN in numerical columns\n    print('----------------------------- kaggle_features_engineering: Impute NaN values for numerical columns with %s method' % p_missing_value_method)\n    numerical_columns_with_nan = [col for col in p.columns if (p[col].dtype == 'int64' or p[col].dtype == 'float64') and p[col].isna().sum() != 0]\n    if len(numerical_columns_with_nan) != 0:\n        print('kaggle_features_engineering: cols_with_missing: %s' % (str(numerical_columns_with_nan)))\n        # Find rows with missing values\n        rows_with_null = p[numerical_columns_with_nan].isnull().any(axis=1)\n        rows_with_missing = p[rows_with_null]\n        print('kaggle_features_engineering: rows_with_missing: %s/%s' % (rows_with_missing.shape[0], p.shape[0]))\n        # Impute missimg values\n        if p_missing_value_method == 'drop_columns': # Impute removing columns\n            p = p.drop(numerical_columns_with_nan, axis = 1)\n        elif p_missing_value_method == 'drop_lines' and len(rows_with_null) != 0: # Impute removing rows\n            p = p.dropna()\n        else: # Imputate using SimpleImputer\n            if Imputer_Instance is None:\n                if p_missing_value_method == 'mean':\n                    Imputer_Instance = SimpleImputer(strategy='mean')\n                elif p_missing_value_method == 'median':\n                    Imputer_Instance = SimpleImputer(strategy='median')\n                else:\n                    raise Exception('kaggle_features_engineering: Invalid method', 'method=%s' % (p_missing_value_method))\n            # else, nothing to do\n            labels = p.columns # Save labels\n            Imputer_Instance = Imputer_Instance.fit(p[numerical_columns_with_nan])\n            p[numerical_columns_with_nan] = Imputer_Instance.transform(p[numerical_columns_with_nan])\n            #p[numerical_columns_with_nan] = pd.DataFrame(Imputer_Instance.fit_transform(p))\n            # Restore column names\n            p.columns = labels\n            print('kaggle_features_engineering: Cleaning NaN values: old_shape: %s / new shape: %s' % (str(old_shape), str(p.shape)))\n            print(p.head())\n            numerical_columns_with_nan = [col for col in p.columns if (p[col].dtype == 'int64' or p[col].dtype == 'float64') and p[col].isna().sum() != 0]\n            if len(numerical_columns_with_nan) != 0:\n                raise Exception('kaggle_features_engineering: There still has numerical columns with NaN value in dataset', 'method=%s' % str(numerical_columns_with_nan))\n    else:\n        print('kaggle_features_engineering: No missing values in numerical columns')\n    print('----------------------------- kaggle_features_engineering: After First round:')\n    #print(p.head())\n    print(p.describe().T)\n\n    print('----------------------------- kaggle_features_engineering: Rebuild lists of categorical and numerical columns')\n    # FIXME Code seems to be useless\n    categorical_columns = [col for col in p.columns if p[col].dtype == 'object']\n    numerical_columns = [col for col in p.columns if p[col].dtype == 'int64' or p[col].dtype == 'float64']\n    print('kaggle_features_engineering: Rebuilt categorical_columns:')\n    print(categorical_columns)\n    print('kaggle_features_engineering: Rebuilt numerical_columns:')\n    print(numerical_columns)\n\n    # Search for categorical variables\n    print('----------------------------- kaggle_features_engineering: Encoding categorical columns:')\n    new_categorical_columns = []\n    if len(categorical_columns) != 0:\n        print('kaggle_features_engineering: categorical_columns: ' + str(categorical_columns))\n        # Compute cardinalities of the categorical vairiables\n        categorical_columns_cardinalities = list(map(lambda col: p[col].nunique(), categorical_columns))\n        print('kaggle_features_engineering: categorical_columns_cardinalities: ')\n        print(categorical_columns_cardinalities)\n        print('kaggle_features_engineering: OneHotEncoder thresholds: %d' % p_categorical_onehot_threshold)\n        # Apply OneHot encoding to categorical value with very low cardinality\n        cols_processed = []\n        new_categorical_columns = categorical_columns.copy()\n        for i in range(0, len(categorical_columns)):\n            if categorical_columns_cardinalities[i] <= p_categorical_onehot_threshold:\n                print('kaggle_features_engineering: OneHotEncoder: %s' % categorical_columns[i])\n                if not Encoders is None:\n                    if not categorical_columns[i] in Encoders:\n                        Encoders[categorical_columns[i]] = OneHotEncoder(handle_unknown = 'ignore', sparse = False)\n                else:\n                    Encoders[categorical_columns[i]] = OneHotEncoder(handle_unknown = 'ignore', sparse = False)\n                new_col = Encoders[categorical_columns[i]].fit_transform(pd.DataFrame(p[categorical_columns[i]]))\n                new_col = pd.DataFrame(new_col, columns = [(categorical_columns[i] + \"_\" + str(j)) for j in range(new_col.shape[1])])\n                new_col.index = p[categorical_columns[i]].index\n                p.drop(categorical_columns[i], inplace = True, axis = 1)\n                p = pd.concat((p, new_col), axis = 1)\n                cols_processed.append(categorical_columns[i])\n                # Update the list of the categorical columns\n                new_categorical_columns.remove(categorical_columns[i])\n                new_categorical_columns.extend(new_col.columns.tolist())\n            else:\n                print('!!!!!!!!!!!!!!!!!!!! kaggle_features_engineering: Cannot process %s' % categorical_columns[i])\n                # Just drop them for the time being\n                # FIXME To be refined using TargetEncoder\n                p.drop(categorical_columns[i], axis = 1, inplace = True)\n                # Update the list of the categorical columns\n                new_categorical_columns.remove(categorical_columns[i])\n        # End of 'for' statement\n        if len(cols_processed) != 0:\n            print('kaggle_features_engineering: Encoders applied on %s' % str(cols_processed))\n            print('kaggle_features_engineering: New datase structure:')\n            #print(p.head())\n            print(p.describe().T)\n            categorical_columns = [col for col in p.columns if p[col].dtype == 'object']\n            print('kaggle_features_engineering: Cleaning categorical values: old_shape: %s / new shape: %s' % (str(old_shape), str(p.shape)))\n            print('kaggle_features_engineering: new Categorical columns:')\n            print(categorical_columns)\n            # Compute new cardinalities of the categorical vairiables\n            categorical_columns_cardinalities = list(map(lambda col: p[col].nunique(), categorical_columns))\n            print('kaggle_features_engineering: New categorical_columns_cardinalities: ')\n            print(categorical_columns_cardinalities)\n        # TODO: Drop categorical variables with extrem cardinalities\n        # Encode categorical variables using numerical mapping\n        for col in categorical_columns:\n            p[col] = Encoder_Instance.fit_transform(p[col].astype(str))\n            # End of 'for' statement\n            print('kaggle_features_engineering: Labelling:')\n            #print(p.head())\n            print(p.describe().T)\n        # End of 'for' statement\n    else:\n        print('kaggle_features_engineering: No categorical values')\n    # Be sure there is no more 'object' types in the dataset\n    categorical_columns = [col for col in p.columns if p[col].dtype == 'object']\n    if len(categorical_columns) != 0:\n        raise Exception('kaggle_features_engineering: There still has object type in dataset', 'method=%s' % str(categorical_columns))\n    print('----------------------------- kaggle_features_engineering: After Second round:')\n    #print(p.head())\n    print(p.describe().T)\n\n    # Identifying & handling outliers\n    print('----------------------------- kaggle_features_engineering: Identifying & handling outliers:')\n    for c in numerical_columns:\n        if EXCLUDE_FROM_OULIERS is None or not c in EXCLUDE_FROM_OULIERS:\n            q_lower, q_upper = np.percentile(p[c], p_outliers_lower_percentile), np.percentile(p[c], p_outliers_upper_percentile)\n            iqr = q_upper - q_lower\n            print('kaggle_features_engineering: IRQ range %f' % iqr)\n            # Outlier cutoff threshold\n            cut_off = iqr * 1.5\n            if not cut_off == 0:\n                lower_bound, upper_bound = q_lower - cut_off, q_upper + cut_off\n                print('kaggle_features_engineering: Outliers thresholds for %s: (%f, %f)' % (c, lower_bound, upper_bound))\n                outliers = [x for x in p[c] if x < lower_bound or x > upper_bound]\n                if p_outliers_impute_method == 'mean':\n                    outliers_impute_method = p[c].mean()\n                else:\n                    outliers_impute_method = p[c].median()\n                if len(outliers) != 0:\n                    print('kaggle_features_engineering: Outliers for %s' % c)\n                    print(outliers)\n                    p[c] = p[c].apply(lambda x: outliers_impute_method if x < lower_bound or x > upper_bound else x)\n                else:\n                    print('No outliers for %s' % c)\n            else:\n                print('No outliers for %s' % c)\n    else:\n        print('kaggle_features_engineering: %s is excluded from Outliers' % c)\n    print('----------------------------- kaggle_features_engineering: After Third round:')\n    #print(p.head())\n    print(p.describe().T)\n    \n    p = kaggle_post_features_engineering(p)\n\n    # Rebuild the list of categorical and numerical features\n    categorical_columns = [col for col in p.columns if p[col].dtype == 'object']\n    numerical_columns = [col for col in p.columns if p[col].dtype == 'int64' or p[col].dtype == 'float64']\n    if p_date_time_columns is not None:\n        if len(categorical_columns) != 0:\n            categorical_columns = list(set(categorical_columns) - set(p_date_time_columns))\n        numerical_columns = list(set(numerical_columns) - set(p_date_time_columns))\n    print('----------------------------- kaggle_features_engineering: Categorical columns after: ', categorical_columns)\n    print('----------------------------- kaggle_features_engineering: Nunmerical columns after: ', numerical_columns)\n\n    print('kaggle_features_engineering: ', list(new_categorical_columns)) \n    print('kaggle_features_engineering: Done') \n    return p, new_categorical_columns, numerical_columns\n    # End of function kaggle_features_engineering","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"TODO: Add data engineering results","metadata":{}},{"cell_type":"markdown","source":"There are different kinds of data transformation:\n- Standardization: It removes the mean and scaling to unit variance of the feature (see point 2.a)\n- Scaling: It rescales the feature values in a range of 0 and 1","metadata":{}},{"cell_type":"code","source":"# b) Data Transforms\nTransform = None\ndef kaggle_data_transform(p_df: pd.core.frame.DataFrame, p_columns:list = None, p_transform: str = 'standard') -> pd.core.frame.DataFrame:\n    \"\"\"\n    Apply data transformation to the provided dataset\n    :parameters p_df: The dataset handle\n    :parameters p_columns: The columns to apply transformation (e.g. we don't apply transformation on categorical column)\n    :parameter p_transform: The type of transormation. Default: 'standard'\n                            'standard': Remove the mean and scaling to unit variance\n                            'scale': Scale feature to a Min/max range\n                            'abs_scale': Scale feature to a range [-1, 1]\n    :return: The dataset after features selection\n    \"\"\"\n    print('----------------------------- kaggle_data_transform -----------------------------')\n    global Transform\n\n    if Transform is None:\n        if p_transform == 'standard':\n            # Standardization, or mean removal and variance scaling\n            Transform = StandardScaler()\n        elif p_transform == 'scale':\n            # Scaling features to a range\n            Transform = MinMaxScaler()\n        elif p_transform == 'abs_scale':\n            # Scaling features to a range\n            Transform = MaxAbsScaler()\n        else:\n            raise Exception('kaggle_data_transform: Wrong parameters', 'p_transform=%s' % p_transform)\n        p = None\n        if p_columns is None: # Apply transformamtion to the whole dataset\n            p = Transform.fit_transform(p_df)\n            p = pd.DataFrame(data = p, columns = p_df.columns)\n        else:\n            p = p_df.copy()\n            p[p_columns] = Transform.fit_transform(p_df[p_columns])\n    else:\n        p = None\n        if p_columns is None: # Apply transformamtion to the whole dataset\n            p = Transform.transform(p_df)\n            p = pd.DataFrame(data = p, columns = p_df.columns)\n        else:\n            p = p_df.copy()\n            p[p_columns] = Transform.fit_transform(p_df[p_columns])\n    print('kaggle_data_transform: Dataset Head:')\n    print(p.head())\n    \n    print('kaggle_data_transform: Done')\n    return p\n    # End of function kaggle_data_transform","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The kaggle_features_selection() function provides:\n- Mutual Information (MI) measures the reduction in uncertainty for one variable given a known value of the other variable. Larger the MI value is, greater is the relationship between the two variables. The MI will help us during the features selection. \nThe MI scores should be compared with the features importances in the model shown by the kaggle_explore_ml() function.\n- Correlation matrix. Based on the threshold values, this function removes features with high level of correlation.","metadata":{}},{"cell_type":"code","source":"# b) Features Selection\ndef kaggle_features_selection(p_df: pd.core.frame.DataFrame, p_correlation_threshold:float = 0.7) -> pd.core.frame.DataFrame:\n    \"\"\"\n    Reorganize the dataset to keep only provided attributes, the target column is the last column of the new dataset\n    :parameters p_df: The dataset handle\n    :parameters p_correlation_threshold: Correlation threshold to calculate lower bound and upper bound for feature removing\n    :return: The dataset after features selection\n    \"\"\"\n    print('----------------------------- kaggle_features_selection -----------------------------')\n    # Mutual Information\n    print('kaggle_visualization: Mutual information scores')\n    show_mutual_information(p_df, p_title = 'Mutual Information Scores');\n    # Build Correlation matrix\n    print('----------------------------- kaggle_features_selection: Correlation table:')\n    p = p_df.copy()\n    p_corr = p.drop([TARGET_COLUMNS], axis = 1)\n    # Extract correlation > 0.7 and < -0.7\n    cor_matrix = p_corr.corr(method = 'pearson')\n    print('----------------------------- kaggle_features_selection: cor_matrix:')\n    print(cor_matrix)\n    upper_tri = cor_matrix.where(np.triu(np.ones(cor_matrix.shape), k = 1).astype(np.bool))\n    print('----------------------------- kaggle_features_selection: Correlations in range > %f and < -%f:' % (p_correlation_threshold, p_correlation_threshold))\n    to_drop = [column for column in upper_tri.columns if any(upper_tri[column] > p_correlation_threshold)]\n    print('kaggle_features_selection: Drop ', to_drop)\n    if len(to_drop) != 0:\n        # Drop correlated columns\n        p.drop(to_drop, axis = 1, inplace = True)\n        print('----------------------------- kaggle_features_selection: new dataset:')\n        print(p.head())\n        print(p.describe().T)\n\n    print('kaggle_features_selection: Done')\n    return p, to_drop \n    # End of function kaggle_features_selection","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Data augmentation is a technic to increase the size of the Training dataset by adding slightly modified copies of already existing data using technics such as flipping (left and/or right), zooming or adding noise. ","metadata":{}},{"cell_type":"code","source":"def kaggle_prepare_dataset(\n                           p_dataset,\n                           p_strategy,\n                           p_training_dataset: bool = False, \n                           p_shuffle_buffer_size: int = 2048, \n                           p_seed: int = SEED_HARCODED_VALUE\n                           ):\n    \"\"\"\n    This function prepares the dataset for model training and validation\n    :parameter p_dataset: The TF dataset to prepare\n    :parameter p_strategy: Are we using TPU/GPU?\n    :parameter p_training_dataset: Set to True if this is a training dataset\n    :parameter p_seed: The seed value for reproductibility\n    Note: Using method dataset.prefetch(tf.data.experimental.AUTOTUNE), \n          this processing happens essentially for free on TPU. \n          Data pipeline code is executed on the \"CPU\" part of the TPU \n          while the TPU itself is computing gradients.\n    \"\"\"\n    print('----------------------------- kaggle_prepare_dataset -----------------------------')\n    if p_training_dataset == True:\n        dataset = p_dataset.repeat() # the training dataset must repeat for several epochs\n        dataset = dataset.shuffle(p_shuffle_buffer_size, seed = p_seed) # Shuffle images of the batch at each iteration\n        dataset = dataset.batch(DL_BATCH_SIZE * p_strategy.num_replicas_in_sync) # Prepare batches for DL\n    else:\n        dataset = p_dataset.batch(DL_BATCH_SIZE * p_strategy.num_replicas_in_sync) # Prepare batches for DL\n        \n    if p_training_dataset == False:\n        dataset = dataset.cache()\n    dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE) # The next batch will always be ready for processing for the next iteration\n    print('kaggle_prepare_dataset: type(dataset) => ', type(dataset))\n    print('kaggle_prepare_dataset: Done')\n    return dataset\n    # End of function kaggle_prepare_dataset","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"TODO Add features selection comments","metadata":{}},{"cell_type":"markdown","source":"After cleaning and transforming the initial dataset, we can use it to train and validate our ML. So, The next step is to split our datasets in two parts:\n- The features, the inputs of the ML/DL\n- The target, the expected output of the ML/DL\n","metadata":{}},{"cell_type":"code","source":"# 4. Evaluate Algorithms\n# a) Split-out validation dataset\ndef kaggle_split_dataset(p_df: pd.core.frame.DataFrame, p_target: str = TARGET_COLUMNS) -> list:\n    \"\"\"\n    Split the into input features and target features\n    :parameters p_df: The dataset handle\n    :parameter p_target The outputs of the Machine Learning\n    :return: The list of input features and target features\n    \"\"\"\n    print('----------------------------- kaggle_split_dataset -----------------------------')\n    y_values = None\n    x_values = None\n    if set([p_target]).issubset(set(p_df.columns)):\n        y_values = p_df[[p_target]]\n        x_values = p_df.drop([p_target], axis = 1)\n    else:\n        x_values = p_df\n    \n    # Re-order column by column name\n    x_values = x_values.reindex(sorted(x_values.columns), axis = 1)\n    \n    print('----------------------------- kaggle_split_dataset: x_values:')\n    print(x_values.head())\n    if not y_values is None:\n        print('----------------------------- kaggle_split_dataset: y_values:')\n        print(y_values.head())\n\n    print('kaggle_split_dataset: Done')\n    return x_values, y_values\n    # End of function kaggle_split_dataset","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can apply different models (linear, non-linear, ensemble...) to build our ML and evaluate their efficiency (4.b)","metadata":{}},{"cell_type":"code","source":"# b.1) Check models\ndef kaggle_check_models(\n                        p_models: list, \n                        p_inputs_training_df: pd.core.frame.DataFrame, \n                        p_outputs_training_df: pd.core.frame.DataFrame, \n                        p_kparts: int = 10, \n                        p_random_state: int = SEED_HARCODED_VALUE, \n                        p_cross_validation: str = 'k-fold', \n                        p_scoring: str = 'accuracy') -> list:\n    \"\"\"\n    Apply different models to train our Machine Learning and evaluate their efficiency\n    :parameter p_models: A list of models to use for to train the Machine Learning\n    :parameter p_inputs_training_df: The training inputs dataset (training attributes)\n    :parameter p_outputs_training_df: The training output dataset (training target)\n    :parameter p_inputs_valid_df: The validation inputs dataset (validation attributes)\n    :parameter p_outputs_valid_df: The validation output dataset (validation target)\n    :parameter p_kparts: The size of the KFolds\n    :parameter p_random_state: \n    :parameter p_cross_validation: KFold or StratifiedKFold. Default: k-fold\n    :parameter p_scoring: \n    :return: The list of couple (result, model name)\n    \"\"\"\n    print('----------------------------- kaggle_check_models -----------------------------')\n    results = []\n    names = []\n    for name, model in p_models:\n        print('kaggle_check_models: Processing %s with type %s' % (name, type(model)))\n        # Create train/test indices to split data in train/test sets\n        if p_cross_validation == 'k-fold':\n            kfold = model_selection.KFold(n_splits = p_kparts, random_state = p_random_state, shuffle = True) # K-fold Cross Validation\n        elif p_cross_validation == 's-k-fold':\n            kfold = model_selection.StratifiedKFold(n_splits = p_kparts, random_state = p_random_state, shuffle = True) # K-fold Cross Validation\n        else:\n            raise Exception('kaggle_check_models: Wrong parameters', 'p_cross_validation:%s' % p_cross_validation)\n        cv_results = None\n        # Evaluate model performance\n        if p_cross_validation == 'k-fold' or p_cross_validation == 's-k-fold':\n            cv_results = model_selection.cross_val_score(model, p_inputs_training_df, p_outputs_training_df, cv = kfold, scoring = p_scoring)\n        else:\n            cv_results = model_selection.cross_val_score(model, p_inputs_training_df, p_outputs_training_df, cv = LeaveOneOut(), scoring = p_scoring)\n        #print('kaggle_check_models: cv_result=%s' % str(cv_results))\n        results.append(cv_results)\n        names.append(name)\n        msg = 'kaggle_check_models: %s metric: %s: %f (+/-%f)' % (p_scoring, name, cv_results.mean(), 2 * cv_results.std())\n        print(msg)\n        # End of 'for' loop\n\n    print('kaggle_check_models: Done')\n    return results, names\n    # End of function kaggle_check_models","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def kaggle_process_raw_predictions(p_predictions, p_labels_range):\n    \"\"\"\n    This function converts raw predictions (outouts of the ML/NN) into a specific format\n    \"\"\"\n    if :\n        \n    # End of function kaggle_process_raw_predictions","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# b.2) Check models for CNN\ndef kaggle_check_dl_model(p_strategy, p_model, p_train_df, p_train_df_size, p_validation_df, p_validation_df_size):\n    \"\"\"\n    \"\"\"\n    print('----------------------------- kaggle_check_dl_model -----------------------------')\n    with p_strategy.scope():\n        print('kaggle_check_dl_model: Setup LearningRateScheduler:')\n        monitor = 'val_loss'\n        lr_callback = None\n        if LR_ALG == 'exp':\n            lr_callback = tf.keras.callbacks.LearningRateScheduler(kaggle_build_learning_rate_schedule, verbose = 2)\n            kaggle_display_learning_rate_curve(kaggle_build_learning_rate_schedule)\n        elif LR_ALG == 'rop':\n            lr_callback = ReduceLROnPlateau(monitor = monitor, factor = 0.2, patience = 8, min_lr = 0.0005)\n        else:\n            raise Exception('kaggle_check_dl_model', 'Wrong learning rate callback')\n        temp_name = next(tempfile._get_candidate_names()) + '.h5'\n        # Training the model\n        print('kaggle_check_dl_model: Train the model:')\n        print('kaggle_check_dl_model: steps_per_epoch = ', p_train_df_size // (DL_BATCH_SIZE * p_strategy.num_replicas_in_sync))\n        # Since the dataset already takes care of batching, batch_size argument is ignored\n        history = p_model.fit(\n            p_train_df,\n            validation_data = p_validation_df,\n            epochs = DL_EPOCH_NUM,\n            steps_per_epoch = p_train_df_size // (DL_BATCH_SIZE * p_strategy.num_replicas_in_sync),\n            callbacks = [lr_callback, ModelCheckpoint(filepath = temp_name, monitor = monitor, save_best_only = True)],\n            verbose = 2\n        )\n        # End of 'with' statement\n\n    # list all data in history\n    print(history.history.keys())\n    # Display leaning curve\n    if 'accuracy' in history.history.keys():\n        # Display leaning curve\n        kaggle_display_learning_curves(\n            history.history['accuracy'],\n            history.history['val_accuracy'],\n            'accuracy',\n            211,\n        )\n    if 'loss' in history.history.keys():\n        kaggle_display_learning_curves(\n            history.history['loss'],\n            history.history['val_loss'],\n            'loss',\n            211,\n        )\n    if 'categorical_accuracy' in history.history.keys():\n        # Display leaning curve\n        kaggle_display_learning_curves(\n            history.history['categorical_accuracy'],\n            history.history['val_categorical_accuracy'],\n            'accuracy',\n            211,\n        )\n    if 'AUC' in history.history.keys():\n        # Display leaning curve\n        kaggle_display_learning_curves(\n            history.history['AUC'],\n            history.history['val_AUC'],\n            'AUC',\n            211,\n        )\n    # Show scores\n    # Split Validation images & labels\n    print('kaggle_check_dl_model: Split images & labels for validation_df')\n    validation_images_df = p_validation_df.map(lambda image, label: image)\n    validation_labels_df = p_validation_df.map(lambda image, label: label).unbatch()\n    # Convert labels into a Pandas dataframe\n    #l = []\n    #for i, e in validation_labels_df.enumerate():\n    #    l[i] = int(e.numpy())\n    #validation_labels_df = pd.DataFrame(l)\n    #print('type(validation_labels_df) ==>', type(validation_labels_df))\n    #print('Main: validation_labels_df.shape = ', len(validation_labels_df), ' / ', validation_df_size)\n    #raise Exception('Stop')\n\n    # Display confusion matrix\n    cm_correct_labels = next(iter(validation_labels_df.batch(p_validation_df_size))).numpy() #np.darray\n    print('kaggle_check_dl_model: cm_correct_labels ==>', cm_correct_labels)\n    cm_probabilities = kaggle_prediction(p_model, validation_images_df)\n    print('kaggle_check_dl_model: cm_probabilities ==>', cm_probabilities)\n    # Convert predictions classes to one hot vectors \n    y_predictions = np.argmax(cm_probabilities, axis = -1)\n    print('kaggle_check_dl_model: y_predictions ==>', y_predictions)\n    labels = range(len(IMAGE_CLASSES))\n    y_predictions = kaggle_process_raw_predictions(y_predictions, labels)\n    # Compute the confusion matrix\n    cmat = confusion_matrix(\n                            cm_correct_labels,\n                            y_predictions,\n                            #FIXME labels = labels,\n                            )\n    # Normalize the confusion matrix\n    cmat = (cmat.T / cmat.sum(axis=1)).T\n    print('kaggle_check_dl_model: cmat = ', cmat)\n    # Display scores\n    score = f1_score(\n        cm_correct_labels,\n        y_predictions,\n        labels=labels,\n        average = 'macro',\n    )\n    print('kaggle_check_dl_model: Score = ', score)\n    precision = precision_score(\n                                cm_correct_labels,\n                                y_predictions,\n                                labels = labels,\n                                average = 'macro',\n    )\n    print('kaggle_check_dl_model: Precision score = ', precision)\n    recall = recall_score(\n                          cm_correct_labels,\n                          y_predictions,\n                          labels = labels,\n                          average='macro',\n    )\n    print('kaggle_check_dl_model: Recall score = ', recall)\n    kaggle_display_confusion_matrix(cmat, score, precision, recall)\n\n    # Retrieve the best weights for the model\n    print('kaggle_check_dl_model: Retrieve the best weights for the model')\n    model = tf.keras.models.load_model(temp_name)\n    os.remove(temp_name)\n\n    print('kaggle_check_dl_model: Done')\n    return (score, precision, recall), history.history, model\n    # End of function kaggle_check_dl_model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, we select the best model based on the scoring (4.c)","metadata":{}},{"cell_type":"code","source":"def kaggle_compare_algorithms_perf(p_names: list, p_metrics: list, p_title: str, p_x_label: str, p_y_label:str) -> int:\n    \"\"\"\n    Compare and return the model with the best results\n    :parameter p_names: The list of models executed\n    :parameter p_metrics: The scorimng obtained by each model\n    :parameter p_title: Performance plot title\n    :parameter p_x_label: Performance plot X-axis label\n    :parameter p_y_label: Performance plot Y-axis label\n    :return: The index of the model with the higher scoring\n    \"\"\"\n    print('----------------------------- kaggle_compare_algorithms_perf -----------------------------')\n    # Extract means & std\n    means = []\n    stds = []\n    for i in range (len(p_names)):\n        cv_results = p_metrics[i]\n        means.append(cv_results.mean())\n        stds.append(cv_results.std())\n        # End of 'for' statement\n    # Display means/standard deviation\n    plt.title(p_title)\n    plt.xlabel(p_x_label)\n    plt.ylabel(p_y_label)\n    plt.errorbar(p_names, means, stds, linestyle='None', marker='^')\n    #plt.savefig('kaggle_algorithms_comparison.png')\n    plt.show()\n    # Select the best algorithm\n    m = np.array(means)\n    maxv = np.amax(m)\n    idx = np.where(m == maxv)[0][0]\n    print('kaggle_compare_algorithms_perf: Max value: %d:%f +/- %f ==> %s' % (idx, maxv, 2 * stds[idx], p_names[idx]))\n\n    print('kaggle_compare_algorithms_perf: Done')\n    return idx\n    # End of function kaggle_compare_algorithms_perf","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, we will use the GridSearchCV() function to find the best model parameters.","metadata":{}},{"cell_type":"code","source":"# 5. Improve Accuracy\n# a) Algorithm Tuning\ndef kaggle_algorithm_tuning(\n                            p_algorithm: list, \n                            p_inputs_training_df: pd.core.frame.DataFrame, \n                            p_outputs_training_df: pd.core.frame.DataFrame, \n                            p_validation_data: list = None,\n                            p_strategy = None, \n                            p_kparts: int = 10, \n                            p_random_state: int = SEED_HARCODED_VALUE\n                            ):\n    \"\"\"\n    Tuning model parameters\n    :parameter p_algorithm: The ML model to tune\n    :parameter p_inputs_training_df: The input training tada\n    :parameter p_outputs_training_df: The target for the training data\n    :parameter p_validation_data: The input validation data\n    :return: The tuned model\n    \"\"\"\n    print('----------------------------- kaggle_algorithm_tuning -----------------------------')\n    model = p_algorithm[1]\n    print('----------------------------- kaggle_algorithm_tuning: %s' % model.__class__.__name__)\n    print('----------------------------- kaggle_algorithm_tuning: model summary:')\n    print(model)\n    print(model.get_params())\n\n    # Fit the model\n    if model.__class__.__name__ == 'LinearRegression': # No Hyper parameters tuning\n        # Hyper parameters tuning\n        print('kaggle_algorithm_tuning: No Hyper parameters tuning for LinearRegression')\n        model.fit(p_inputs_training_df, p_outputs_training_df)\n        return model\n    elif not model.__class__.__name__.startswith('Keras'):\n        # Hyper parameters tuning\n        print('----------------------------- kaggle_algorithm_tuning: Hyper parameters tuning')\n        if model.__class__.__name__.startswith('SV'):\n            param_grid = {\n                'C': [0.1, 1, 10, 100], \n                'gamma': [1, 0.1, 0.01, 0.001],\n                'kernel': ['rbf', 'poly', 'sigmoid']\n            }\n        elif model.__class__.__name__ == 'LinearDiscriminantAnalysis':\n            param_grid = {\n                'shrinkage': [ 'auto', 0, 0.5, 1.0 ],\n                'solver': [ 'svd', 'lsqr', 'eigen' ], \n                'tol': [1.0e-4, 1.0e-3, 1.0e-2, 1.0e-1]\n            }\n        elif model.__class__.__name__.startswith('LogisticRegression'):\n            param_grid = {\n                'C': [0.1, 1, 10, 100], \n                'penalty': ['l1','l2'],\n                'solver': ['newton-cg', 'lbfgs', 'liblinear', 'sag'],\n                'multi_class': ['ovr', 'multinomial'],\n                'class_weight': ['None', 'balanced'],\n                'max_iter': [100, 150, 200], \n                'tol': [1.0e-4, 1.0e-3, 1.0e-2, 1.0e-1]\n            }\n        elif model.__class__.__name__.startswith('KNeighbors'):\n            param_grid = {\n                'n_neighbors': [4, 8, 16, 32], \n                'weights': ['uniform', 'distance'],\n                'algorithm': ['ball_tree', 'kd_tree', 'brute']\n            }\n        elif model.__class__.__name__.startswith('LGBM'):\n            param_grid = {\n                'num_leaves': [32, 128],\n                'reg_alpha': [0.1, 0.5],\n                'min_data_in_leaf': [32, 64, 128, 256],\n                'lambda_l1': [0, 1, 1.5],\n                'lambda_l2': [0, 1],\n                'learning_rate': [0.01, 0.05, 0.1, 0.2, 0.4, 0.6],\n            }\n        else:\n            # Global grid parameters\n            param_grid = {\n                'n_estimators': [512, 1024, 2048],\n                'max_depth': [16, 24 , 32],\n                'max_leaf_nodes': [64, 128, 256],\n                'learning_rate': [0.01, 0.05, 0.1, 0.2, 0.4, 0.6],\n                #'loss': ['deviance'],\n                #'min_samples_split': np.linspace(0.1, 0.5, 3),\n                #'min_samples_leaf': np.linspace(0.1, 0.5, 3),\n                #'max_features': ['log2', 'sqrt'],\n                #'criterion': ['friedman_mse',  'mae'],\n                #'subsample': np.linspace(0.5, 1.0, 3),\n            }\n            # Remove unsupported parameters\n            if model.__class__.__name__.startswith('RandomForest'):\n                del param_grid['learning_rate']\n\n        cv = model_selection.RepeatedStratifiedKFold(n_splits = p_kparts, n_repeats = 10, random_state = p_random_state) #5 for hardcoded value\n        tunning = GridSearchCV(\n            estimator = model,\n            param_grid = param_grid, \n            cv = cv, \n            n_jobs = 5, \n            scoring = 'neg_mean_squared_error',\n            verbose = 2\n        )\n        model = tunning.fit(p_inputs_training_df, p_outputs_training_df)\n    elif model.__class__.__name__.startswith('Keras'):\n        with p_strategy.scope():\n            param_grid = {\n                #'optimizer': [ 'rmsprop', 'adam' ],\n                #'init': [ 'glorot_uniform' , 'normal' , 'uniform' ],\n                'nb_epoch': [48, 64],\n                'batch_size': [ 48 * p_strategy.num_replicas_in_sync, 64 * p_strategy.num_replicas_in_sync ]\n            }\n            cv = model_selection.RepeatedStratifiedKFold(n_splits = p_kparts, n_repeats = 10, random_state = p_random_state) #5 for hardcoded value\n            tunning = GridSearchCV(estimator = model, param_grid = param_grid, cv = cv, n_jobs = 1, verbose = 2) # Cannot use n_jobs = 5 (-1) because sklean does not support pickle\n                                                                                                                 # See https://stackoverflow.com/questions/48717451/keras-kerasclassifier-gridsearch-typeerror-cant-pickle-thread-lock-objects\n            early_stopping = EarlyStopping(\n                min_delta = 0.001, # minimium amount of change to count as an improvement\n                patience = 20, # how many epochs to wait before stopping\n                restore_best_weights = True,\n            )\n            model = tunning.fit(p_inputs_training_df, \n                                p_outputs_training_df, \n                                validation_data = p_validation_data, \n                                epochs = DL_EPOCH_NUM, \n                                batch_size = DL_BATCH_SIZE * p_strategy.num_replicas_in_sync, \n                                callbacks = [early_stopping]\n                               )\n    else:\n        raise Exception('kaggle_algorithm_tuning: Wrong parameters combination', 'DL with p_strategy set to None')\n\n    print('----------------------------- kaggle_algorithms_tuning: Hyper parameters tuning ended:')\n    print('kaggle_algorithm_tuning: Hyper parameters tuning: best_score_=', model.best_score_)\n    print('kaggle_algorithm_tuning: Hyper parameters tuning: best_params_=', model.best_params_)\n    print('kaggle_algorithm_tuning: Hyper parameters tuning: best_estimator_=', model.best_estimator_)\n    for mean, std, params in zip(model.cv_results_['mean_test_score'], model.cv_results_['std_test_score'], model.cv_results_['params']):\n        print(\"%0.3f (+/-%0.03f) for %r\" % (mean, std * 2, params))\n    model = model.best_estimator_\n    \n    print('kaggle_algorithm_tuning: Done')\n    return model\n    # End of function kaggle_algorithm_tuning","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we reached the point where we can evaluate our model with Validation and/or Test datasets.","metadata":{}},{"cell_type":"code","source":"# b) Ensembles\n# 6. Finalize Model\n# a) Predictions on validation dataset\ndef kaggle_validation_prediction(p_model, p_inputs, p_expected_outputs) -> np.ndarray:\n    \"\"\"\n    Executes prediction (p_inputs) and compares outputs against expected outputs (Validation) using the specified ML model\n    :parameter p_model: \n    :parameter p_inputs: \n    :parameter p_expected_outputs: \n    \"\"\"\n    print('----------------------------- kaggle_validation_prediction -----------------------------')\n    print('kaggle_validation_prediction: model=%s - is_class:%s - is_regr:%s' % (p_model.__class__.__name__, str(is_classifier(p_model)), str(is_regressor(p_model))))\n    y_predictions = p_model.predict(p_inputs)\n    print('kaggle_validation_prediction: Score: ', round(p_model.score(p_inputs, p_expected_outputs) * 100, 2), \" %\")\n    if is_regressor(p_model) or p_model.__class__.__name__ == 'KerasRegressor': # Regression metrics (continuous target values)\n        # Get scoring\n        get_scoring(p_expected_outputs, y_predictions)\n        # Analyze residual errors\n        plt.scatter(p_expected_outputs, y_predictions)\n        plt.show()\n        # TODO Interpreting the Cofficients if possible\n    elif is_classifier(p_model) or p_model.__class__.__name__ == 'KerasClassifier': # Classification metrics (class target values)\n        # Get scoring\n        get_scoring(p_expected_outputs, y_predictions)\n    else:\n        raise Exception('kaggle_validation_prediction: Invalid model')\n    print('kaggle_validation_prediction: prediction is %s' % (str(y_predictions)))\n\n    print('kaggle_validation_prediction: Done')\n    return y_predictions\n    # End of function kaggle_validation_prediction\n\ndef kaggle_prediction(p_model, p_inputs) -> np.ndarray:\n    \"\"\"\n    Executes prediction (p_inputs) using the specified ML model\n    :parameter p_model: \n    :parameter p_inputs:  \n    \"\"\"\n    print('----------------------------- kaggle_prediction -----------------------------')\n    y_prediction = p_model.predict(p_inputs)\n    print('kaggle_prediction: prediction is %s' %(str(y_prediction)))\n    print('kaggle_prediction: Done')\n    return y_prediction\n# b) Create standalone model on entire training dataset\n# TODO","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The functions below are some helpers to save or reload the model and features engineering data.","metadata":{}},{"cell_type":"code","source":"# Pickling Keras Models to prevent error 'TypeError: can't pickle weakref objects' while saving the model\n# http://zachmoshe.com/2017/04/03/pickling-keras-models.html\ndef make_keras_picklable():\n    \"\"\"\n    This function is used to pickling Keras Models to prevent error 'TypeError: can't pickle weakref objects' while saving the model\n    See http://zachmoshe.com/2017/04/03/pickling-keras-models.html\n    \"\"\"\n    def __getstate__(self):\n        model_str = \"\"\n        with tempfile.NamedTemporaryFile(suffix='.hdf5', delete=True) as fd:\n            keras.models.save_model(self, fd.name, overwrite=True)\n            model_str = fd.read()\n        d = { 'model_str': model_str }\n        return d\n\n    def __setstate__(self, state):\n        with tempfile.NamedTemporaryFile(suffix='.hdf5', delete=True) as fd:\n            fd.write(state['model_str'])\n            fd.flush()\n            model = keras.models.load_model(fd.name)\n        self.__dict__ = model.__dict__\n\n\n    cls = keras.models.Model\n    cls.__getstate__ = __getstate__\n    cls.__setstate__ = __setstate__","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# c) Save model for later use\ndef kaggle_save_model(p_model, p_paths: str, p_file_name:str) -> None:\n    \"\"\"\n    Save the provided model in binary/pickle format and the Encoders/Imputers\n    :parameter p_model: The ML model to save\n    :parameter p_paths: The path to save the files, ending with a '/' (e.g. ./)\n    :parameter p_file_name: The file name woithout extension file (e.g. './MyModel')\n    \"\"\"\n    global Encoders, Encoder_Instance, Imputer_Instance, Transform\n\n    print('----------------------------- kaggle_save_model -----------------------------')\n    make_keras_picklable()\n    # Serialize the model\n    pickle.dump(p_model, open(p_paths + p_file_name + '.pkl', 'wb'))\n    print('kaggle_save_model: Done: %s' % (p_file_name + '.pkl'))\n    # Save Encoders, Encoder_Instance and Imputer_Instance\n    if not Encoders is None and len(Encoders) != 0:\n        pickle.dump(Encoders, open(p_paths + 'Encoders.pkl', 'wb'))\n    if not Encoder_Instance is None:\n        pickle.dump(Encoder_Instance, open(p_paths + 'Encoder_Instance.pkl', 'wb'))\n    if not Imputer_Instance is None:\n        pickle.dump(Imputer_Instance, open(p_paths + 'Imputer_Instance.pkl', 'wb'))\n    if not Transform is None:\n        pickle.dump(Transform, open(p_paths + 'Transform.pkl', 'wb'))\n\n    print('kaggle_save_model: Done')\n    # End of function kaggle_save_model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load model\ndef kaggle_load_model(p_paths: str, p_file_name:str):\n    \"\"\"\n    Load a model in binary/pickle format and the Encoders/Imputers\n    :parameter p_model: The ML model to save\n    :parameter p_paths: The path to save the files, ending with a '/' (e.g. ./)\n    :parameter p_file_name: The file name woithout extension file (e.g. './MyModel')\n    \"\"\"\n    global Encoders, Encoder_Instance, Imputer_Instance, Transform\n\n    print('----------------------------- kaggle_load_model -----------------------------')\n    try:\n        Encoders = pickle.load(open(p_paths + 'Encoders.pkl', 'rb'))\n    except:\n        Encoders = dict()\n    try:\n        Encoder_Instance = pickle.load(open(p_paths + 'Encoder_Instance.pkl', 'rb'))\n    except:\n        Encoder_Instance = LabelEncoder()\n    try:\n        Imputer_Instance = pickle.load(open(p_paths + 'Imputer_Instance.pkl', 'rb'))\n    except:\n        Imputer_Instance = None\n    try:\n        Transform = pickle.load(open(p_paths + 'Transform.pkl', 'rb'))\n    except:\n        Transform = None\n    model = pickle.load(open(p_paths + p_file_name + '.pkl', 'rb'))\n    \n    print('kaggle_load_model: Done')\n    return model\n    # End of function kaggle_load_model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The function kaggle_explore_ml() provides some insights from our ML.\n\n# PIMA diabetes\nFor PIMA India, if we cmpare the Mutual Information scores with the most important feature, the Body mass index feature seems to be main key of the diabete.\n","metadata":{}},{"cell_type":"code","source":"def kaggle_explore_ml(p_model, p_x_validation: pd.core.frame.DataFrame, p_y_validation: pd.core.frame.DataFrame, p_random_state: int = SEED_HARCODED_VALUE) -> None:\n    \"\"\"\n    Apply feature importance concept to our ML \n    :parameter p_model: The predictions to save\n    \"\"\"\n    print('----------------------------- kaggle_explore_ml -----------------------------')\n    result = permutation_importance(p_model, p_x_validation, p_y_validation, n_repeats = 32, random_state = p_random_state)\n    sorted_idx = result.importances_mean.argsort()\n    print('----------------------------- kaggle_explore_ml: result:')\n    print(sorted_idx)\n\n    fig, ax = plt.subplots()\n    ax.boxplot(result.importances[sorted_idx].T, vert = False, labels = p_x_validation.columns[sorted_idx])\n    ax.set_title(\"Permutation Importances (Validation set)\")\n    fig.tight_layout()\n    plt.show()\n    print('kaggle_explore_ml: Done')\n    # End of function kaggle_explore_ml","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The function kaggle_save_result() saves prediction results in Kaggle format for submission to Kaggle Compete","metadata":{}},{"cell_type":"code","source":"def kaggle_save_result(p_model, p_column:str, p_predictions: np.ndarray, p_test_file_name: str, p_file_name:str) -> None:\n    \"\"\"\n    Save prediction results in Kaggle format for submission to compete\n    :parameter p_model: The predictions to save\n    :parameter p_column: The index column name\n    :parameter p_predictions: The prediction results based on Test dataset\n    :parameter p_test_file_name: The original validation dataset, to extract the index for Kaggle submission (see p_column)\n    :parameter p_file_name: The file name without extension file (e.g. './MyResults.csv')\n    \"\"\"\n    print('----------------------------- kaggle_save_result -----------------------------')\n    # Reload PassengerID list\n    validation_df = pd.read_csv(p_test_file_name)\n    p = validation_df[[p_column]].astype(int)\n    # FIXME How to proceed with multiple outputs?\n    # Write the submission file\n    print(p.shape)\n    print(p_predictions.shape)\n    print(len(p_predictions.squeeze()))\n    pred = pd.DataFrame({p_column: list(p.squeeze()), TARGET_COLUMNS: p_predictions.astype(int).squeeze()})\n    pred.to_csv(p_file_name, index = False)\n    print('kaggle_save_result: Done: ', p_file_name)\n    # End of function kaggle_save_model\n\ndef kaggle_save_result_dl(p_model, p_columns:list, p_predictions: np.ndarray, p_test_df, test_df_size: int, p_file_name:str) -> None:\n    \"\"\"\n    Save prediction results in Kaggle format for submission to compete based on Images processing\n    :parameter p_model: The predictions to save\n    :parameter p_column: The list of the columns name\n    :parameter p_predictions: The prediction results based on Test dataset\n    :parameter p_test_df: The test dataset containing unlabelled images\n    :parameter test_df_size: The size of the test dataset (# of images)\n    :parameter p_file_name: The file name without extension file (e.g. './MyResults.csv')\n    \"\"\"\n    print('----------------------------- kaggle_save_result_dl -----------------------------')\n    print('p_predictions ==>', p_predictions)\n    print(p_predictions.shape)\n    print(len(p_predictions.squeeze()))\n    # Get image ids from test set and convert to unicode\n    test_ids_ds = p_test_df.map(lambda image, idnum: idnum).unbatch()\n    test_ids = next(iter(test_ids_ds.batch(test_df_size))).numpy().astype('U')\n    print('type(test_ids) ==>', type(test_ids))\n    print('test_ids ==>', test_ids)\n    # Write the submission file\n    pred = pd.DataFrame({p_columns[0]: list(test_ids.squeeze()), p_columns[1]: p_predictions.astype(int).squeeze()})\n    pred.to_csv(p_file_name, index = False)\n    print('kaggle_save_result_dl: Done: ', p_file_name)\n    # End of function kaggle_save_result_dl","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Data augmentation is a technic to increase the size of the Training dataset by adding slightly modified copies of already existing data using technics such as flipping (left and/or right), zooming or adding noise.\n\nFinaly, here is the entry point function and the main call:","metadata":{}},{"cell_type":"code","source":"def kaggle_main() -> None:\n    global DL_INPUT_SHAPE\n    \n    # Set defaults\n    kaggle_set_seed()\n    kaggle_set_mp_default()\n    \n    # Current path\n    print(os.path.abspath(os.getcwd()))\n    # Kaggle current path and files\n    #for dirname, _, filenames in os.walk('/kaggle/input'):\n    #    for filename in filenames:\n    #        print(os.path.join(dirname, filename))\n\n    # Modules version\n    kaggle_modules_versions()\n\n    flags = FLAGS\n    # Parse arguments. Used only if this notebook code is used as a standalone Python script\n    #parser = argparse.ArgumentParser()\n    #parser.add_argument('--summarize', help = 'Process statistical analyze', action='store_true')\n    #parser.add_argument('--summarize-only', help = 'Process only statistical analyze', action='store_true')\n    #parser.add_argument('--visualize', help = 'Generate different plots based on statistical analyze', action='store_true')\n    #parser.add_argument('--no-data-cleaning', help = 'Do not apply Data Cleaning', action='store_true')\n    #parser.add_argument('--neural-network', help = 'Use neural network as ML', action='store_true')\n    #args = parser.parse_args()\n    #if args.summarize or args.summarize_only:\n    #    flags |= ExecutionFlags.DATA_STAT_SUMMURIZE_FLAG\n    #if args.visualize:\n    #    flags |= ExecutionFlags.DATA_VISUALIZATION_FLAG\n    #if args.no_data_cleaning:\n    #    flags |= ~ExecutionFlags.DATA_CLEANING_FLAG\n    #if args.neural_network:\n    #    flags |= ExecutionFlags.USE_NEURAL_NETWORK_FLAG\n    \n    # TODO Uncomment if using Pima Indians diabetes dataset\n    #flags &= ~ExecutionFlags.DATA_CLEANING_FLAG\n    print('Main: Generic template approach to ''play'' with the Machine Learning concepts: flags=%s' % str(flags))\n    print('Main: Playgrounf name: ', ML_NAME)\n\n    kaggle_pre_main()\n\n    strategy, global_path = kaggle_tpu_detection()\n    if global_path is None: # Force global_path to local path when datasets not copied to gs://kds-xxx\n        global_path = os.path.join(os.path.abspath(os.getcwd()), '../input')\n        global_path = os.path.join(global_path, DATABASE_NAME)\n\n    if not flags & ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG:\n        y_test_df = None\n        if ML_NAME == 'Pima':\n            train_df, validation_df, test_df, y_test_df = kaggle_load_datasets(\n                                                                               p_url = DATABASE_URI,\n                                                                               p_labels = COLUMNS_LABEL\n                                                                               )\n        elif ML_NAME == 'Iris':\n            train_df, validation_df, test_df, y_test_df = kaggle_load_datasets(\n                                                                               p_url = DATABASE_URI,\n                                                                               p_labels = COLUMNS_LABEL\n                                                                               )\n        elif ML_NAME == 'Titanic':\n            train_df, validation_df, test_df, _ = kaggle_load_datasets(\n                                                                       p_url = None, \n                                                                       p_train_path = '../input/titanic/train.csv', \n                                                                       p_validation_path = '../input/titanic/test.csv'\n                                                                       )\n        else:\n            raise Exception('A dataset shall be selected')\n        print('Main: training dataset')\n        print(train_df.head())\n        print('Main: validation dataset')\n        print(validation_df.head())\n        print('Main: test dataset')\n        print(test_df.head())\n        print('Main: test outputs dataset is %s ' % ('empty' if y_test_df is None else 'not null'))\n        # Do a basic ML evaluation as reference for later result comparisons\n        if not flags & ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG:\n            kaggle_ml_quick_and_dirty(train_df, validation_df, test_df, p_test_outputs_df = y_test_df)\n        else:\n            kaggle_dl_quick_and_dirty(train_df, validation_df, strategy, test_df, p_test_outputs_df = y_test_df)\n    else:\n        # We are loading Images datasets\n        if ML_NAME == 'FlowerClassification':\n            # Images databse already in Tfrec format\n            image_folder = 'tfrecords-jpeg-' + str(IMAGE_NUM_PIXELS) + 'x' + str(IMAGE_NUM_PIXELS)\n            train_df, validation_df, test_df, train_df_size, validation_df_size, test_df_size = kaggle_load_images_datasets(p_train_url = None,\n                                                                                                                            p_train_path = image_folder + '/train/*.tfrec',\n                                                                                                                            p_validation_path = image_folder + '/val/*.tfrec',\n                                                                                                                            p_global_path = global_path,\n                                                                                                                            p_test_path = image_folder + '/test/*.tfrec'\n                                                                                                                            )\n        elif ML_NAME == 'BMS-MolecularTranslation':\n            # Images and lables are separated\n            train_df, validation_df, test_df, train_df_size, validation_df_size, test_df_size = kaggle_load_images_datasets(p_train_url = DATABASE_NAME,\n                                                                                                                            p_labels = COLUMNS_LABEL, \n                                                                                                                            p_global_path = global_path, \n                                                                                                                            p_test_url = DATABASE_NAME\n                                                                                                                            )\n        elif ML_NAME == 'HumanProteinAtlas':\n            # Images and lables are separated\n            train_df, validation_df, test_df, train_df_size, validation_df_size, test_df_size = kaggle_load_images_datasets(p_train_url = DATABASE_NAME,\n                                                                                                                            p_labels = COLUMNS_LABEL, \n                                                                                                                            p_global_path = global_path,\n                                                                                                                            p_test_url = DATABASE_NAME\n                                                                                                                            )\n        else:\n            raise Exception('A dataset shall be selected')\n        print('type(train_df) ==> ', type(train_df))\n        print('Main: train_df_size', train_df_size)\n        print('type(validation_df) ==> ', type(validation_df))\n        print('Main: validation_df_size', validation_df_size)\n        print('type(test_df_size) ==> ', type(train_df))\n        print('Main: test_df_size', test_df_size)\n    # End of datasets loading operation\n\n#    raise Exception('Stop')\n\n    # Data visualization\n    if not flags & ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG:\n        numerical_columns = None\n        categorical_columns = None\n        if flags & ExecutionFlags.DATA_STAT_SUMMURIZE_FLAG == ExecutionFlags.DATA_STAT_SUMMURIZE_FLAG:\n            print('Main: Summurize data for training dataset')\n            kaggle_summurize_data(train_df)\n            print('Main: Summurize data for validation dataset')\n            kaggle_summurize_data(validation_df)\n            print('Main: Summurize data for test dataset')\n            kaggle_summurize_data(test_df)\n        #    if args.summarize_only:\n        #        return\n\n        # Data visualization\n        if flags & ExecutionFlags.DATA_VISUALIZATION_FLAG == ExecutionFlags.DATA_VISUALIZATION_FLAG:\n            print('Main: Visualisation for train dataset')\n            kaggle_visualization(train_df)\n            cross_dataset_visualization([train_df, validation_df])\n    else:\n        # Data visualization\n        if flags & ExecutionFlags.DATA_VISUALIZATION_FLAG == ExecutionFlags.DATA_VISUALIZATION_FLAG:\n            print(\"Main: Training images:\", train_df)\n            kaggle_image_visualization(train_df)\n            print (\"Main: Validation images:\")\n            kaggle_image_visualization(validation_df)\n            print(\"Main: Test images:\")\n            kaggle_image_visualization(test_df)\n    # End of Data visualization\n\n#    raise Exception('Stop')\n\n    # Data engineering\n    if not flags & ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG:\n        categorical_features = None\n        numerical_features = None\n        if flags & ExecutionFlags.DATA_CLEANING_FLAG == ExecutionFlags.DATA_CLEANING_FLAG:\n            train_df, new_categorical_features, numerical_features = kaggle_features_engineering(train_df, p_date_time_columns = DATE_TIME_COLUMNS, p_missing_value_method = 'mean')\n            print(train_df.columns)\n            validation_df, _, _ = kaggle_features_engineering(validation_df, p_date_time_columns = DATE_TIME_COLUMNS, p_missing_value_method = 'mean')\n            print(validation_df.columns)\n            if len(train_df.columns) != len(validation_df.columns):\n                l = list(set(train_df.columns) - set(validation_df.columns))\n                print('Main: Features unaligned for validation_df: ', l)\n                validation_df[l] = 0\n                print(validation_df.columns)\n            test_df, _, _ = kaggle_features_engineering(test_df, p_date_time_columns = DATE_TIME_COLUMNS, p_missing_value_method = 'mean')\n            print(test_df.columns)\n            if len(train_df.columns) != len(test_df.columns):\n                l = list(set(train_df.columns) - set(test_df.columns) - set([TARGET_COLUMNS]))\n                print('Main: Features unaligned for test_df: ', l)\n                test_df[l] = 0\n                print(test_df.columns)\n            print('Main: training dataset after data engineering')\n            print(train_df.head())\n            print('Main: validation dataset after data engineering')\n            print(validation_df.head())\n            print('Main: test dataset after data engineering')\n            print(test_df.head())\n            # Visualization after data engineering\n            if flags & ExecutionFlags.DATA_STAT_SUMMURIZE_FLAG == ExecutionFlags.DATA_STAT_SUMMURIZE_FLAG:\n                print('Main: Summurize data for training dataset')\n                kaggle_summurize_data(train_df)\n                print('Main: Summurize data for validation dataset')\n                kaggle_summurize_data(validation_df)\n                print('Main: Summurize data for test dataset')\n                kaggle_summurize_data(test_df)\n            # Do a basic ML evaluation as reference for the end\n            if not flags & ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG:\n                kaggle_ml_quick_and_dirty(train_df, validation_df, test_df, p_test_outputs_df = y_test_df)\n            else:\n                kaggle_dl_quick_and_dirty(train_df, validation_df, strategy, test_df, p_test_outputs_df = y_test_df)\n\n        if flags & ExecutionFlags.DATA_TRANSFORM_FLAG == ExecutionFlags.DATA_TRANSFORM_FLAG:\n            # Extract non  categorical columns based on categorical_column list\n            if not numerical_features is None:\n                columns_to_transform = numerical_features\n                if not NON_TRANSFORMABLE_COLUMNS is None:\n                    columns_to_transform = list(set(columns_to_transform) - set(NON_TRANSFORMABLE_COLUMNS))\n                if not OUTPUT_IS_REGRESSION:\n                    # Remove TARGET_COLUMNS from columns_to_transform list because it is a classification (yes, no)\n                    columns_to_transform = list(set(columns_to_transform) - set([TARGET_COLUMNS]))\n                print('Main: columns_to_transform: %s' % str(columns_to_transform))\n                print('Main: columns_to_transform: ')\n                print(train_df.describe())\n                train_df = kaggle_data_transform(train_df, columns_to_transform, p_transform = 'standard')\n                validation_df = kaggle_data_transform(validation_df, columns_to_transform, p_transform = 'standard')\n                if OUTPUT_IS_REGRESSION:\n                    # Remove TARGET_COLUMNS from columns_to_transform list because test_df does not contain targets\n                    columns_to_transform = list(set(columns_to_transform) - set([TARGET_COLUMNS]))\n                test_df = kaggle_data_transform(test_df, columns_to_transform, p_transform = 'standard')\n                print('Main: training dataset after features transformation')\n                print(train_df.head())\n                print('Main: validation dataset after features transformation')\n                print(validation_df.head())\n                print('Main: test dataset after features transformation')\n                print(test_df.head())\n                # Do a basic DL evaluation as reference for the end\n                if not flags & ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG:\n                    kaggle_ml_quick_and_dirty(train_df, validation_df, test_df, p_test_outputs_df = y_test_df)\n                else:\n                    kaggle_dl_quick_and_dirty(train_df, validation_df, strategy, test_df, p_test_outputs_df = y_test_df)\n\n#    raise Exception('Stop')\n\n        train_df, dropped_features = kaggle_features_selection(train_df)\n        #dropped_features = []\n        if len(dropped_features) != 0:\n            if set(dropped_features).issubset(set(validation_df.columns)):\n                validation_df.drop(dropped_features, inplace = True, axis = 1)\n            if set(dropped_features).issubset(set(test_df.columns)):\n                test_df.drop(dropped_features, inplace = True, axis = 1)\n            print('Main: training dataset after features selection')\n            print(train_df.head())\n            print('Main: validation dataset after features selection')\n            print(validation_df.head())\n            print('Main: test dataset after features selection')\n            print(test_df.head())\n    else:\n        # 1. Prepare the datasets for model training, validation and tets\n        train_df = kaggle_prepare_dataset(train_df, strategy, True)\n        # Inspect Validation dataset\n        #for image_batch, label_batch in train_df.take(1):\n        #    pass\n        #print('Main: Train data label inspection:', label_batch.numpy())\n        #print(image_batch.numpy().shape, len(image_batch.numpy()), label_batch.numpy().shape)\n        validation_df = kaggle_prepare_dataset(validation_df, strategy)\n        # Inspect Validation dataset\n        #for image_batch, label_batch in validation_df.take(1):\n        #    pass\n        #print('Main: Validation data label inspection:', label_batch.numpy())\n        #print(image_batch.numpy().shape, len(image_batch.numpy()), label_batch.numpy().shape)\n        \n        test_df = kaggle_prepare_dataset(test_df, strategy)\n        #test_df = test_df.batch(DL_BATCH_SIZE * strategy.num_replicas_in_sync) # Prepare batches for DL\n        #test_df = test_df.prefetch(tf.data.experimental.AUTOTUNE) # The next batch will always be ready for processing for the next iteration\n        # Inspect test dataset\n        #for image_batch in test_df.take(1):\n        #    pass\n        #print('Main: Test inspection:')\n        #print('type(image_batch) ==> ', type(image_batch))\n        #print('len(image_batch) ==> ', len(image_batch))\n        #print(image_batch)\n        print('Main: Dataset: %d training images, %d validation images, %d unlabeled test images' % (train_df_size, validation_df_size, test_df_size))\n        # 2. Keep a copy of the original validation_df for model evaluation\n        copy_validation_df = validation_df\n        # 3. Do a basic DL evaluation as reference for the end, without data augmentation\n        print('----------------------------- kaggle_dl_quick_and_dirty -----------------------------')\n        # FIXME Enhance function kaggle_dl_quick_and_dirty to support CNN and more\n        model = None\n        if OUTPUT_IS_REGRESSION: # Use regression algorithms\n            with strategy.scope():\n                pass # TODO\n                # End of 'with' statement\n        else:\n            with strategy.scope():\n                model = kaggle_create_sequential_classifier_model(\n                                                                  #strategy\n                                                                  #p_input_shape = DL_INPUT_SHAPE, \n                                                                  #p_drop_rate = DL_DROP_RATE,\n                                                                  p_loss = 'categorical_crossentropy', \n                                                                  p_metrics = ['categorical_accuracy', 'accuracy'],\n                                                                  p_pretrained_layers = ['VGG19'], \n                                                                  p_image_size = IMAGE_SIZE, \n                                                                  p_class_num = len(IMAGE_CLASSES),\n                                                                  p_augmentations = None,\n                                                                  )\n        with strategy.scope():\n            results = model.evaluate(validation_df, verbose = 2)#, steps = 1\n            print(\"test loss, test acc:\", results)\n            # End of 'with' statement\n    # End of Data engineering    \n\n#    raise Exception('Stop')\n\n    # Building models and training operation\n    models = []\n    if not flags & ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG:\n        ml_inputs_training_df, ml_outputs_training_df = kaggle_split_dataset(train_df)\n        ml_inputs_validation_df, ml_outputs_validation_df = kaggle_split_dataset(validation_df)\n        ml_inputs_test_df, _ = kaggle_split_dataset(test_df)\n        DL_INPUT_SHAPE = [ml_inputs_training_df.shape[1]]\n        print('Main: ml_inputs_training_df')\n        print(ml_inputs_training_df.head())\n        print('Main: ml_outputs_training_df')\n        print(ml_outputs_training_df.head())\n        print('Main: ml_inputs_validation_df')\n        print(ml_inputs_validation_df.head())\n        print('Main: ml_outputs_validation_df')\n        print(ml_outputs_validation_df.head())\n        print('Main: ml_inputs_test_df')\n        print(ml_inputs_test_df.head())\n\n        # Checking models\n        scoring = None\n        if OUTPUT_IS_REGRESSION: # Use regression algorithms\n            scoring = 'r2' # 'r2' or 'neg_mean_absolute_error'\n            if not flags & ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG:\n                models.append(('LR', linear_model.LinearRegression()))\n                models.append(('SGDC', linear_model.SGDRegressor(random_state = SEED_HARCODED_VALUE)))\n                models.append(('LASSO', linear_model.Lasso()))\n                models.append(('EN', linear_model.ElasticNet()))\n                models.append(('KNN', neighbors.KNeighborsRegressor(n_neighbors = 8)))\n                models.append(('CART', tree.DecisionTreeRegressor(max_leaf_nodes = 256, random_state = SEED_HARCODED_VALUE)))\n                models.append(('LGBMR', lgb.LGBMRegressor(n_estimators = 1024, num_leaves = 128, max_depth = 32, learning_rate=0.05, random_state = SEED_HARCODED_VALUE)))\n                models.append(('XGB', xgb.XGBRegressor(n_estimators = 1024, learning_rate=0.05, random_state = SEED_HARCODED_VALUE)))\n                models.append(('BGK', ensemble.GradientBoostingRegressor(n_estimators = 1024, max_depth = 32, random_state = SEED_HARCODED_VALUE)))\n                models.append(('RF', ensemble.RandomForestRegressor(n_estimators = 1024, max_depth = 32, max_features = 4, random_state = SEED_HARCODED_VALUE)))\n                models.append(('SVR', svm.SVR()))\n            if flags & ExecutionFlags.USE_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_NEURAL_NETWORK_FLAG or flags & ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG:\n                models.append(('NNR', KerasRegressor(build_fn = kaggle_create_sequential_regressor_model, nb_epoch = DL_EPOCH_NUM, batch_size = DL_BATCH_SIZE * strategy.num_replicas_in_sync, verbose = 1)))\n            # TODO Add support of regressors!\n        else: # Use classifier algorithms\n            scoring = 'accuracy'\n            if not flags & ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG:\n                models.append(('LR', linear_model.LogisticRegression(random_state = SEED_HARCODED_VALUE)))\n                models.append(('SGDC', linear_model.SGDClassifier(random_state = SEED_HARCODED_VALUE)))\n                models.append(('LDA', discriminant_analysis.LinearDiscriminantAnalysis()))\n                models.append(('KNN', neighbors.KNeighborsClassifier(n_neighbors = 8)))\n                models.append(('CART', tree.DecisionTreeClassifier(max_leaf_nodes = 256, random_state = SEED_HARCODED_VALUE)))\n                models.append(('LGBMC', lgb.LGBMClassifier(n_estimators = 1024, num_leaves = 128, max_depth = 32, learning_rate=0.05, random_state = SEED_HARCODED_VALUE)))\n                models.append(('XGB', xgb.XGBClassifier(n_estimators = 1024, learning_rate=0.05, random_state = SEED_HARCODED_VALUE)))\n                models.append(('BGK', ensemble.GradientBoostingClassifier(n_estimators = 1024, max_depth = 32, random_state = SEED_HARCODED_VALUE)))\n                models.append(('RF', ensemble.RandomForestClassifier(n_estimators = 1024, max_depth = 32, max_features = 4, random_state = SEED_HARCODED_VALUE)))\n                models.append(('NB', naive_bayes.GaussianNB()))\n                models.append(('SVC', svm.SVC(random_state = SEED_HARCODED_VALUE)))\n            if flags & ExecutionFlags.USE_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_NEURAL_NETWORK_FLAG or flags & ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_ONLY_NEURAL_NETWORK_FLAG:\n                models.append(('NNC', KerasClassifier(build_fn = kaggle_create_sequential_classifier_model, nb_epoch = DL_EPOCH_NUM, batch_size = DL_BATCH_SIZE * strategy.num_replicas_in_sync, verbose = 1)))\n                # TODO Add support of classifiers!\n\n        # Check models\n        results, names = kaggle_check_models(models, ml_inputs_training_df, ml_outputs_training_df, p_cross_validation = 's-k-fold', p_scoring = scoring)\n        best_alg = kaggle_compare_algorithms_perf(names, results, 'Algorithms Comparison', 'Algorithms', 'Accuracy')\n        ml = kaggle_algorithm_tuning(models[best_alg], ml_inputs_training_df, ml_outputs_training_df, (ml_inputs_validation_df, ml_outputs_validation_df), p_strategy = strategy)\n    else:\n        if OUTPUT_IS_REGRESSION: # Use regression algorithms\n            with strategy.scope():\n                pass # TODO\n            # End of 'with' statement\n        else:\n            # Data augmentation is part of the model.  Input images will only be augmented during calls to model.fit (not model.evaluate or model.predict) \n            # See https://www.tensorflow.org/tutorials/images/data_augmentation\n            with strategy.scope():\n                models.append(('VGG19', kaggle_create_sequential_classifier_model( # FIXME Use parameters for loss and metrics\n                                                                                  #strategy\n                                                                                  #p_input_shape = DL_INPUT_SHAPE, \n                                                                                  #p_drop_rate = DL_DROP_RATE,\n                                                                                  p_loss = 'categorical_crossentropy', \n                                                                                  p_metrics = ['categorical_accuracy', 'accuracy'],\n                                                                                  p_pretrained_layers = ['VGG19'], \n                                                                                  p_image_size = IMAGE_SIZE, \n                                                                                  p_class_num = len(IMAGE_CLASSES),\n                                                                                  p_augmentations = ['flip_right_left', 'flip_up_down'], #'rotate', 'random_brightness'],\n                                                                                  )))\n                models.append(('DenseNet201', kaggle_create_sequential_classifier_model( # FIXME Use parameters for loss and metrics\n                                                                                     #strategy\n                                                                                     #p_input_shape = DL_INPUT_SHAPE, \n                                                                                     #p_drop_rate = DL_DROP_RATE,\n                                                                                     p_loss = 'categorical_crossentropy', \n                                                                                     p_metrics = ['categorical_accuracy', 'accuracy'],\n                                                                                     p_pretrained_layers = ['DenseNet201'], \n                                                                                     p_image_size = IMAGE_SIZE, \n                                                                                     p_class_num = len(IMAGE_CLASSES),\n                                                                                     p_augmentations = ['flip_right_left', 'flip_up_down'], #'rotate', 'random_brightness'],\n                                                                                     )))\n        with strategy.scope():\n            print('----------------------------- kaggle_check_dl_models -----------------------------')\n            results = []\n            histories = []\n            trained_models = []\n            for name, model in models:\n                print('kaggle_check_dl_models: Processing %s with type %s' % (name, type(model)))\n                result, history, trained_model = kaggle_check_dl_model(strategy, model, train_df, train_df_size, validation_df, validation_df_size)\n                results.append(result)\n                histories.append(history)\n                trained_models.append(trained_model)\n                print('kaggle_check_dl_models: result: ', result)\n                #print('kaggle_check_dl_models: history: ', history)\n                #break # For debug\n                # End of 'for' statement\n            print('kaggle_check_dl_models: After loop, results: ', results)\n            ml = trained_models[1]\n            # TODO Add support of regressor!\n        # End of 'with' statement\n    # End of building models and training operation\n\n    # Evaluate the model with the validation dataset\n    if not flags & ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG:\n        y_predictions = kaggle_validation_prediction(ml, ml_inputs_validation_df, ml_outputs_validation_df)\n        kaggle_explore_ml(ml, ml_inputs_validation_df, y_predictions)\n    else:\n        dataset = copy_validation_df.unbatch().batch(10)\n        batch = iter(dataset)\n        images, labels = next(batch)\n        probabilities = kaggle_prediction(ml, images)\n        y_predictions = np.argmax(probabilities, axis=-1)\n        kaggle_display_batch((images, labels), y_predictions)\n\n    # Test the model with unseen images\n    if not flags & ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG:\n        y_predictions = kaggle_prediction(ml, ml_inputs_test_df)\n        kaggle_explore_ml(ml, ml_inputs_test_df, y_predictions)\n    else:\n        images_df = test_df.map(lambda image, idnum: image)\n        probabilities = kaggle_prediction(ml, images_df)\n        y_predictions = np.argmax(probabilities, axis = -1)\n        print(y_predictions)\n\n    # Save the result for the Kaggle compete\n    print('Main: Save Kaggel compete submission')\n    if not flags & ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG:\n        if not TEST_FILE_NAME is None:\n            kaggle_save_result(ml, 'id', y_predictions, TEST_FILE_NAME, '/kaggle/working/submission.csv')\n        else:\n            print('Main: TEST_FILE_NAME is None, cannot save Kaggle compete submission')\n    else:\n        kaggle_save_result_dl(ml, ['id', 'label'], y_predictions, test_df, test_df_size, '/kaggle/working/submission.csv')\n        \n    # Save the model and try to reload it\n    if not flags & ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG == ExecutionFlags.USE_CNN_NEURAL_NETWORK_FLAG:\n        print('Main: Save the model')\n        file_name = ml.__class__.__name__\n        kaggle_save_model(ml, '/kaggle/working/', file_name)\n        print('Main: Reload the model')\n        ml = kaggle_load_model('/kaggle/working/', file_name)\n        y_predictions = kaggle_validation_prediction(ml, ml_inputs_validation_df, ml_outputs_validation_df)\n        kaggle_explore_ml(ml, ml_inputs_validation_df, y_predictions)\n\n    kaggle_post_main()\n\n    print('Main: End of processing')\n    # End of function kaggle_main","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ouf, now, we can execute all the sequences described above and get some results:","metadata":{}},{"cell_type":"code","source":"# Entry point\nprint(\"Starting at \", datetime.now().strftime(\"%d/%m/%Y %H:%M:%S\"))\nkaggle_main()\nprint(\"Ending at \", datetime.now().strftime(\"%d/%m/%Y %H:%M:%S\"))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***If you liked this Notebook, please upvote.\nGives Motivation to make new Notebooks :)***","metadata":{}}]}