{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"# Hi to all! In this notebook, I have prepared an example of how to use the KerasTuner API(https://keras.io/api/keras_tuner/)\n# to find model hyperparameters.\n# I took this wonderful notebook as a basis: https://www.kaggle.com/code/roberthatch/gislr-lb-0-63-on-the-shoulders\n# In version 17, I added the code from this notebook: https://www.kaggle.com/code/dschettler8845/gislr-how-to-ensemble\n# So let's start","metadata":{}},{"cell_type":"markdown","source":"# Since version 13, I use StratifiedKFold.","metadata":{}},{"cell_type":"markdown","source":"# GISLR [LB 0.63]: On the Shoulders of Giants\nThis notebook is built on top of the great work from the community! In particular, this notebook is copied directly from Darien's excellent work [here][1], and the features are loaded from my notebook [here][2], and that notebook is directly based off of the format [here][3].\n\nThis notebook is going public to celebrate my first ever public LB first place! Fun!\n\nIn this notebook, the main updates are:\n1. Adding the time dimension! Finally, right? There's a 0.58 notebook [here][4] that STILL is just flat mean + std, so I hardly think 0.63 is nearing the limit of what we can achieve. Check out the [features notebook][2], I do two different techniques for the time dimension, resize is one, and padding into a multiple of 3, and then taking mean and std of each third of the data is the other. It took some doing to implement them within the TFLite format/restrictions, so I hope it serves as a useful resource for others!\n2. Just small hyperparameter adjustments.\n3. Oh, and drop face, drop pose, add lips. But refer to the features notebook for the precise details.\n\n# More Credits\nThanks to many Kagglers who have shared ideas. Lonnie's notebook [here][5] helped me and pretty much everyone as we were struggling to figure out this TFLite Model competition format. Andrew helped give suggestions on some of my questions on the discussion page.\n\n[1]: https://www.kaggle.com/code/dschettler8845/gislr-learn-eda-baseline\n[2]: https://www.kaggle.com/code/roberthatch/gislr-feature-data-on-the-shoulders\n[3]: https://www.kaggle.com/code/mayukh18/gislr-feature-data\n[4]: https://www.kaggle.com/code/medali1992/gislr-nn-arcface-baseline\n[5]: https://www.kaggle.com/code/lonnieqin/isolated-sign-language-recognition-with-dnn","metadata":{}},{"cell_type":"markdown","source":"<br><br>\n# MOTIVATION\n**\\#OpenToWork**\n\nAfter 9 years as a Silicon Firmware Engineer at Intel, I am now looking for my next career opportunity. Naturally I am especially interested in Machine Learning opportunities.\n\nIn this notebook, my goals are to:\n* Demonstrate my core strengths. Soft skills like problem solving, communication, and passion.\n* Build up my knowledge skills. I love machine learning and, in particular, love to optimize anything and everything. In this notebook, I explore TensorFlow and Neural Networks.\n* Continue to build on previous Kaggle successes.\n\n**LinkedIn:** https://www.linkedin.com/in/robhatch/\n\n**Kaggle:** https://www.kaggle.com/roberthatch","metadata":{}},{"cell_type":"code","source":"import os\nos.environ['TF_DETERMINISTIC_OPS'] = '1'\nos.environ['TF_CUDNN_DETERMINISTIC'] = '1'","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:24:16.372693Z","iopub.execute_input":"2023-03-12T19:24:16.373106Z","iopub.status.idle":"2023-03-12T19:24:16.396803Z","shell.execute_reply.started":"2023-03-12T19:24:16.373069Z","shell.execute_reply":"2023-03-12T19:24:16.395731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# IMPORTS","metadata":{}},{"cell_type":"code","source":"print(\"\\n... PIP INSTALLS STARTING ...\\n\")\n!pip install -q --upgrade tensorflow-io\ntry:\n    import mediapipe as mp\nexcept:\n    !pip install -q mediapipe\n    import mediapipe as mp\nprint(\"\\n... PIP INSTALLS COMPLETE ...\\n\")\n\n!pip install lion-tf\nprint(\"\\n... IMPORTS STARTING ...\\n\")\nprint(\"\\n\\tVERSION INFORMATION\")\n# Competition Specific Imports (You'll see why we need these later)\n# mediapipe above\n\n# Machine Learning and Data Science Imports (basics)\nimport tensorflow as tf; print(f\"\\t\\t– TENSORFLOW VERSION: {tf.__version__}\");\nimport tensorflow_io as tfio; print(f\"\\t\\t– TENSORFLOW-IO VERSION: {tfio.__version__}\");\nimport pandas as pd; pd.options.mode.chained_assignment = None; pd.set_option('display.max_columns', None);\nimport numpy as np; print(f\"\\t\\t– NUMPY VERSION: {np.__version__}\");\nimport sklearn; print(f\"\\t\\t– SKLEARN VERSION: {sklearn.__version__}\");\n\n# Built-In Imports (mostly don't worry about these)\nfrom kaggle_datasets import KaggleDatasets\nfrom collections import Counter\nfrom datetime import datetime\nfrom zipfile import ZipFile\nfrom glob import glob\nimport Levenshtein\nimport warnings\nimport requests\nimport hashlib\nimport imageio\nimport IPython\nimport sklearn\nimport urllib\nimport zipfile\nimport pickle\nimport random\nimport shutil\nimport string\nimport json\nimport math\nimport time\nimport gzip\nimport ast\nimport sys\nimport io\nimport os\nimport gc\nimport re\n\ndef seed_it_all(seed=7):\n    \"\"\" Attempt to be Reproducible \"\"\"\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    random.seed(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n\nseed_it_all()\n\nprint(\"\\n\\n... IMPORTS COMPLETE ...\\n\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-03-12T19:24:16.399002Z","iopub.execute_input":"2023-03-12T19:24:16.399387Z","iopub.status.idle":"2023-03-12T19:25:08.922508Z","shell.execute_reply.started":"2023-03-12T19:24:16.399349Z","shell.execute_reply":"2023-03-12T19:25:08.921237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# SETUP","metadata":{}},{"cell_type":"markdown","source":"### HELPER FUNCTIONS","metadata":{}},{"cell_type":"code","source":"def read_json_file(file_path):\n    try:\n        # Open the file and load the JSON data into a Python object\n        with open(file_path, 'r') as file:\n            json_data = json.load(file)\n        return json_data\n    except FileNotFoundError:\n        # Raise an error if the file path does not exist\n        raise FileNotFoundError(f\"File not found: {file_path}\")\n    except ValueError:\n        # Raise an error if the file does not contain valid JSON data\n        raise ValueError(f\"Invalid JSON data in file: {file_path}\")\n\n\nROWS_PER_FRAME = 543  # number of landmarks per frame\ndef load_relevant_data_subset(pq_path):\n    data_columns = ['x', 'y', 'z']\n    data = pd.read_parquet(pq_path, columns=data_columns)\n    n_frames = int(len(data) / ROWS_PER_FRAME)\n    data = data.values.reshape(n_frames, ROWS_PER_FRAME, len(data_columns))\n    return data.astype(np.float32)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:08.925216Z","iopub.execute_input":"2023-03-12T19:25:08.926134Z","iopub.status.idle":"2023-03-12T19:25:08.934484Z","shell.execute_reply.started":"2023-03-12T19:25:08.926091Z","shell.execute_reply":"2023-03-12T19:25:08.933434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### LOAD DATA","metadata":{}},{"cell_type":"code","source":"# Define the path to the root data directory\nDATA_DIR         = \"/kaggle/input/asl-signs\"\n\nprint(\"\\n... BASIC DATA SETUP STARTING ...\\n\")\nprint(\"\\n\\n... LOAD TRAIN DATAFRAME FROM CSV FILE ...\\n\")\n\ntrain_df = pd.read_csv(os.path.join(DATA_DIR, \"train.csv\"))\ntrain_df[\"path\"] = DATA_DIR+\"/\"+train_df[\"path\"]\ndisplay(train_df)\n\nprint(\"\\n\\n... LOAD SIGN TO PREDICTION INDEX MAP FROM JSON FILE ...\\n\")\ns2p_map = {k.lower():v for k,v in read_json_file(os.path.join(DATA_DIR, \"sign_to_prediction_index_map.json\")).items()}\np2s_map = {v:k for k,v in read_json_file(os.path.join(DATA_DIR, \"sign_to_prediction_index_map.json\")).items()}\nencoder = lambda x: s2p_map.get(x.lower())\ndecoder = lambda x: p2s_map.get(x)\nprint(s2p_map)\n","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:08.936164Z","iopub.execute_input":"2023-03-12T19:25:08.936836Z","iopub.status.idle":"2023-03-12T19:25:09.174974Z","shell.execute_reply.started":"2023-03-12T19:25:08.936798Z","shell.execute_reply":"2023-03-12T19:25:09.174007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PREPROCESSING\n\nThe preprocessing cells need to match the same cells from the feature generation notebook","metadata":{}},{"cell_type":"markdown","source":"## Configuration","metadata":{}},{"cell_type":"code","source":"DROP_Z = False\n\nNUM_FRAMES = 15\nSEGMENTS = 3\n\nLEFT_HAND_OFFSET = 468\nPOSE_OFFSET = LEFT_HAND_OFFSET+21\nRIGHT_HAND_OFFSET = POSE_OFFSET+33\n\n## average over the entire face, and the entire 'pose'\naveraging_sets = [[0, 468], [POSE_OFFSET, 33]]\n\nlip_landmarks = [61, 185, 40, 39, 37,  0, 267, 269, 270, 409,\n                 291,146, 91,181, 84, 17, 314, 405, 321, 375, \n                 78, 191, 80, 81, 82, 13, 312, 311, 310, 415, \n                 95, 88, 178, 87, 14,317, 402, 318, 324, 308]\nleft_hand_landmarks = list(range(LEFT_HAND_OFFSET, LEFT_HAND_OFFSET+21))\nright_hand_landmarks = list(range(RIGHT_HAND_OFFSET, RIGHT_HAND_OFFSET+21))\n\npoint_landmarks = [item for sublist in [lip_landmarks, left_hand_landmarks, right_hand_landmarks] for item in sublist]\n\nLANDMARKS = len(point_landmarks) + len(averaging_sets)\nprint(LANDMARKS)\nif DROP_Z:\n    INPUT_SHAPE = (NUM_FRAMES,LANDMARKS*2)\nelse:\n    INPUT_SHAPE = (NUM_FRAMES,LANDMARKS*3)\n\nFLAT_INPUT_SHAPE = (INPUT_SHAPE[0] + 2 * (SEGMENTS + 1)) * INPUT_SHAPE[1]\n    ","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:09.177622Z","iopub.execute_input":"2023-03-12T19:25:09.178681Z","iopub.status.idle":"2023-03-12T19:25:09.190257Z","shell.execute_reply.started":"2023-03-12T19:25:09.178639Z","shell.execute_reply":"2023-03-12T19:25:09.189182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helper Functions","metadata":{}},{"cell_type":"code","source":"def tf_nan_mean(x, axis=0):\n    return tf.reduce_sum(tf.where(tf.math.is_nan(x), tf.zeros_like(x), x), axis=axis) / tf.reduce_sum(tf.where(tf.math.is_nan(x), tf.zeros_like(x), tf.ones_like(x)), axis=axis)\n\ndef tf_nan_std(x, axis=0):\n    d = x - tf_nan_mean(x, axis=axis)\n    return tf.math.sqrt(tf_nan_mean(d * d, axis=axis))\n\ndef flatten_means_and_stds(x, axis=0):\n    # Get means and stds\n    x_mean = tf_nan_mean(x, axis=0)\n    x_std  = tf_nan_std(x,  axis=0)\n\n    x_out = tf.concat([x_mean, x_std], axis=0)\n    x_out = tf.reshape(x_out, (1, INPUT_SHAPE[1]*2))\n    x_out = tf.where(tf.math.is_finite(x_out), x_out, tf.zeros_like(x_out))\n    return x_out\n","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:09.191803Z","iopub.execute_input":"2023-03-12T19:25:09.192254Z","iopub.status.idle":"2023-03-12T19:25:09.203844Z","shell.execute_reply.started":"2023-03-12T19:25:09.192217Z","shell.execute_reply":"2023-03-12T19:25:09.202841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TensorFlow Feature Preprocessing Layer","metadata":{}},{"cell_type":"code","source":"class FeatureGen(tf.keras.layers.Layer):\n    def __init__(self):\n        super(FeatureGen, self).__init__()\n    \n    def call(self, x_in):\n        if DROP_Z:\n            x_in = x_in[:, :, 0:2]\n        x_list = [tf.expand_dims(tf_nan_mean(x_in[:, av_set[0]:av_set[0]+av_set[1], :], axis=1), axis=1) for av_set in averaging_sets]\n        x_list.append(tf.gather(x_in, point_landmarks, axis=1))\n        x = tf.concat(x_list, 1)\n\n        x_padded = x\n        for i in range(SEGMENTS):\n            p0 = tf.where( ((tf.shape(x_padded)[0] % SEGMENTS) > 0) & ((i % 2) != 0) , 1, 0)\n            p1 = tf.where( ((tf.shape(x_padded)[0] % SEGMENTS) > 0) & ((i % 2) == 0) , 1, 0)\n            paddings = [[p0, p1], [0, 0], [0, 0]]\n            x_padded = tf.pad(x_padded, paddings, mode=\"SYMMETRIC\")\n        x_list = tf.split(x_padded, SEGMENTS)\n        x_list = [flatten_means_and_stds(_x, axis=0) for _x in x_list]\n\n        x_list.append(flatten_means_and_stds(x, axis=0))\n        \n        ## Resize only dimension 0. Resize can't handle nan, so replace nan with that dimension's avg value to reduce impact.\n        x = tf.image.resize(tf.where(tf.math.is_finite(x), x, tf_nan_mean(x, axis=0)), [NUM_FRAMES, LANDMARKS])\n        x = tf.reshape(x, (1, INPUT_SHAPE[0]*INPUT_SHAPE[1]))\n        x = tf.where(tf.math.is_nan(x), tf.zeros_like(x), x)\n        x_list.append(x)\n        x = tf.concat(x_list, axis=1)\n        return x\n\nprint(FeatureGen()(tf.keras.Input((543, 3), dtype=tf.float32, name=\"inputs\")))\nFeatureGen()(load_relevant_data_subset(train_df.path[0]))","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:09.205437Z","iopub.execute_input":"2023-03-12T19:25:09.205880Z","iopub.status.idle":"2023-03-12T19:25:12.400147Z","shell.execute_reply.started":"2023-03-12T19:25:09.205843Z","shell.execute_reply":"2023-03-12T19:25:12.398942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TRAIN","metadata":{}},{"cell_type":"code","source":"## Hyperparameters\nBATCH_SIZE = 128\nVAL_PCT = 0.1\nLEARNING_RATE = 0.000333\nLR_PATIENCE = 2\nLR_REDUCTION_FACTOR = 0.8\nEPOCHS = 100\n\nSTARTING_LAYER_SIZE = 1024\nDROPOUTS = [0.4, 0.4]\n","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:12.401578Z","iopub.execute_input":"2023-03-12T19:25:12.402561Z","iopub.status.idle":"2023-03-12T19:25:12.408686Z","shell.execute_reply.started":"2023-03-12T19:25:12.402511Z","shell.execute_reply":"2023-03-12T19:25:12.407170Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_x    = np.load(\"/kaggle/input/gislr-feature-data-on-the-shoulders/feature_data.npy\").astype(np.float32)\ntrain_y    = np.load(\"/kaggle/input/gislr-feature-data-on-the-shoulders/feature_labels.npy\").astype(np.uint8)\nprint(train_x.shape, train_y.shape)\nif DROP_Z:\n    train_x = np.reshape(train_x, [train_x.shape[0], -1, 3])\n    train_x = train_x[:, :, 0:2]\n    train_x = np.reshape(train_x, [train_x.shape[0], -1])\n    print(train_x.shape, train_y.shape)\n\nN_TOTAL = train_x.shape[0]\nFLAT_FRAME_SHAPE = train_x.shape[1]\nassert(FLAT_FRAME_SHAPE == FLAT_INPUT_SHAPE)\nN_VAL   = int(N_TOTAL*VAL_PCT)\nN_TRAIN = N_TOTAL-N_VAL\n\n#random_idxs = random.sample(range(N_TOTAL), N_TOTAL)\n#train_idxs, val_idxs = np.array(random_idxs[:N_TRAIN]), np.array(random_idxs[N_TRAIN:])\n\nfrom sklearn.model_selection import StratifiedKFold\nskf = StratifiedKFold(n_splits=5)\nfold=0\nfor i, (train_index, test_index) in enumerate(skf.split(train_x,train_y)):\n    print(f\"Fold {i}:\")\n    if i==fold:\n        #print(train_index)\n        val_x_fold, val_y_fold = train_x[test_index], train_y[test_index]\n        train_x_fold, train_y_fold = train_x[train_index], train_y[train_index]\n\ntrain_x_fold.shape\n","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:12.409987Z","iopub.execute_input":"2023-03-12T19:25:12.410322Z","iopub.status.idle":"2023-03-12T19:25:46.675361Z","shell.execute_reply.started":"2023-03-12T19:25:12.410276Z","shell.execute_reply":"2023-03-12T19:25:46.674223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Importing kerastuner.tuners","metadata":{}},{"cell_type":"code","source":"\nfrom kerastuner.tuners import RandomSearch, BayesianOptimization\nfrom lion_tf import Lion","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:46.677035Z","iopub.execute_input":"2023-03-12T19:25:46.677705Z","iopub.status.idle":"2023-03-12T19:25:46.787631Z","shell.execute_reply.started":"2023-03-12T19:25:46.677663Z","shell.execute_reply":"2023-03-12T19:25:46.786547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def fc_block(inputs, output_channels, dropout=0.2, activation=\"gelu\"):\n    x = tf.keras.layers.Dense(output_channels)(inputs)\n    x = tf.keras.layers.BatchNormalization()(x)\n    x = tf.keras.layers.Activation(activation)(x)\n    x = tf.keras.layers.Dropout(dropout)(x)\n    return x\ndef build_model(hp):\n    \n    \n    #Here we set the hyperparameters for the search, this is just a basic example:    \n    STARTING_LAYER_SIZE = hp.Int(name=\"STARTING_LAYER_SIZE\", min_value=1426, max_value=1554, step=32)\n    dropout = [hp.Float(name=\"dropout1\", min_value=0.24, max_value=0.40, step=0.02),   \n               hp.Float(name=\"dropout2\", min_value=0.24, max_value=0.40, step=0.02)]\n    DROPOUTS_len= 2#hp.Int(name=\"dropouts_len\", min_value=2, max_value=5, step=1)\n    hp_activation = hp.Choice('hp_activation', values=[\"gelu\", \"relu\"])\n    hp_learning_rate = hp.Choice('hp_learning_rate', values=[LEARNING_RATE,LEARNING_RATE*0.7, LEARNING_RATE*0.9])\n    \n    hp_optimizer = hp.Choice('optimizer', values=['adam'])\n\n    if hp_optimizer == 'lion':\n        optimizer = Lion(hp_learning_rate)\n    elif hp_optimizer == 'adam':\n        optimizer = tf.keras.optimizers.Adam(hp_learning_rate)\n    def get_model(n_labels=250, init_fc=STARTING_LAYER_SIZE, flat_frame_len=FLAT_FRAME_SHAPE):\n        _inputs = tf.keras.layers.Input(shape=(flat_frame_len,))\n        x = _inputs\n\n        # Define layers\n        \n        \n        for i in range(DROPOUTS_len):\n            x = fc_block(\n                x, output_channels=init_fc//(2**i), \n                dropout=dropout[i]\n            )\n\n        # Define output layer\n        _outputs = tf.keras.layers.Dense(n_labels, activation=\"softmax\")(x)\n\n        # Build the model\n        model = tf.keras.models.Model(inputs=_inputs, outputs=_outputs)\n        return model\n\n    model = get_model()\n    model.compile(optimizer, \"sparse_categorical_crossentropy\", metrics=\"acc\")\n    model.summary()\n    return model\n    #tf.keras.utils.plot_model(model)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:46.792242Z","iopub.execute_input":"2023-03-12T19:25:46.793124Z","iopub.status.idle":"2023-03-12T19:25:46.805129Z","shell.execute_reply.started":"2023-03-12T19:25:46.793082Z","shell.execute_reply":"2023-03-12T19:25:46.803632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#strategy = tf.distribute.MirroredStrategy()\ntuner = BayesianOptimization(\n    build_model,\n    objective='val_acc',\n    max_trials=20,\n    executions_per_trial=1,\n    directory='/',\n    project_name='asl',\n    #distribution_strategy=strategy\n)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:46.806762Z","iopub.execute_input":"2023-03-12T19:25:46.807149Z","iopub.status.idle":"2023-03-12T19:25:46.970503Z","shell.execute_reply.started":"2023-03-12T19:25:46.807113Z","shell.execute_reply":"2023-03-12T19:25:46.969672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!mkdir models\ncb_list = [\n    tf.keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True, verbose=1),\n    tf.keras.callbacks.ReduceLROnPlateau(patience=LR_PATIENCE, factor=LR_REDUCTION_FACTOR, verbose=1)\n]\n#history = model.fit(train_x, train_y, validation_data=(val_x, val_y), epochs=EPOCHS, callbacks=cb_list, batch_size=BATCH_SIZE)\n#model.save(\"./models/asl_model\")","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:46.971621Z","iopub.execute_input":"2023-03-12T19:25:46.972003Z","iopub.status.idle":"2023-03-12T19:25:48.016226Z","shell.execute_reply.started":"2023-03-12T19:25:46.971945Z","shell.execute_reply":"2023-03-12T19:25:48.014780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#We are looking for hyperparameters. Note that I only use 10 epochs, the number of epochs can be increased\nsearch=False\nif search:\n    tuner.search(train_x_fold, train_y_fold, epochs=100, validation_data=(val_x_fold, val_y_fold), callbacks=cb_list, batch_size=BATCH_SIZE)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:48.018144Z","iopub.execute_input":"2023-03-12T19:25:48.018677Z","iopub.status.idle":"2023-03-12T19:25:48.025137Z","shell.execute_reply.started":"2023-03-12T19:25:48.018629Z","shell.execute_reply":"2023-03-12T19:25:48.023988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if search:\n    best_hps = tuner.get_best_hyperparameters()[0].values\n    print('The best hyperparameters:',best_hps)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:48.026767Z","iopub.execute_input":"2023-03-12T19:25:48.027883Z","iopub.status.idle":"2023-03-12T19:25:48.036570Z","shell.execute_reply.started":"2023-03-12T19:25:48.027844Z","shell.execute_reply":"2023-03-12T19:25:48.035348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if search:\n    STARTING_LAYER_SIZE = best_hps['STARTING_LAYER_SIZE']\n    dropout = [best_hps['dropout1'],best_hps['dropout2']]\n    DROPOUTS_len= 2#best_hps['dropouts_len']\n    hp_activation=best_hps['hp_activation']\n    hp_learning_rate=best_hps['hp_learning_rate']\n    hp_optimizer=best_hps['optimizer']\nelse:\n    #0.7616174221038818\n    STARTING_LAYER_SIZE = 1426#1024\n    dropout = [0.3,0.3]\n    DROPOUTS_len= 2#3\n    hp_activation='relu'#'gelu'\n    hp_learning_rate=0.000333#LEARNING_RATE\n    hp_optimizer='adam'#'adam'\n    \n    \nif hp_optimizer == 'lion':\n        optimizer = Lion(hp_learning_rate)\nelif hp_optimizer == 'adam':\n    optimizer = tf.keras.optimizers.Adam(hp_learning_rate)    \n    \ndef get_model(n_labels=250, init_fc=STARTING_LAYER_SIZE, flat_frame_len=FLAT_FRAME_SHAPE):\n    _inputs = tf.keras.layers.Input(shape=(flat_frame_len,))\n    x = _inputs\n\n    # Define layers\n\n    for i in range(DROPOUTS_len):\n        x = fc_block(\n            x, output_channels=init_fc//(2**i), \n            dropout=dropout[i]\n        )\n    # Define output layer\n    _outputs = tf.keras.layers.Dense(n_labels, activation=\"softmax\")(x)\n\n    # Build the model\n    model = tf.keras.models.Model(inputs=_inputs, outputs=_outputs)\n    return model\n\nmodel = get_model()\nmodel.compile(optimizer, \"sparse_categorical_crossentropy\", metrics=\"acc\")\nmodel.summary()\ntf.keras.utils.plot_model(model)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:48.038282Z","iopub.execute_input":"2023-03-12T19:25:48.039579Z","iopub.status.idle":"2023-03-12T19:25:48.519199Z","shell.execute_reply.started":"2023-03-12T19:25:48.039539Z","shell.execute_reply":"2023-03-12T19:25:48.518032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if_train=False\nimport gc\ngc.collect()\nfolds=[0,1,2,3,4]\nmodels=[]\nfor fold in folds:\n    print(f\"Fold {fold}:\")\n    cb_list = [\n    tf.keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True, verbose=1),\n    tf.keras.callbacks.ReduceLROnPlateau(patience=LR_PATIENCE, factor=LR_REDUCTION_FACTOR, verbose=1)]\n    for i, (train_index, test_index) in enumerate(skf.split(train_x,train_y)):\n        if i==fold:\n            val_x_fold, val_y_fold = train_x[test_index], train_y[test_index]\n            train_x_fold, train_y_fold = train_x[train_index], train_y[train_index]\n        \n    model = get_model()\n    if hp_optimizer == 'lion':\n        optimizer = Lion(hp_learning_rate)\n    elif hp_optimizer == 'adam':\n        optimizer = tf.keras.optimizers.Adam(hp_learning_rate)    \n    model.compile(optimizer, \"sparse_categorical_crossentropy\", metrics=\"acc\")\n    #model.summary()\n    #tf.keras.utils.plot_model(model)\n    if if_train:\n        history = model.fit(train_x_fold, train_y_fold, validation_data=(val_x_fold, val_y_fold), epochs=EPOCHS, callbacks=cb_list, batch_size=BATCH_SIZE)#\n    else:\n        model = tf.keras.models.load_model(f'/kaggle/input/gislr-tf-on-the-shoulders-kerastunerversion14/models/asl_model_{fold}')#The models are trained in version 14\n        model.evaluate(val_x_fold, val_y_fold)\n    model.save(f\"./models/asl_model_{fold}\")\n    \n    #model.evaluate(val_x_fold, val_y_fold)\n    #for x,y in zip(val_x_fold[:10], val_y_fold[:10]):\n    #    print(f\"PRED: {decoder(np.argmax(model.predict(tf.expand_dims(x, axis=0), verbose=0), axis=-1)[0]):<20} – GT: {decoder(y)}\")\n    models.append(model)\n    gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:25:48.521111Z","iopub.execute_input":"2023-03-12T19:25:48.521410Z","iopub.status.idle":"2023-03-12T19:26:36.110700Z","shell.execute_reply.started":"2023-03-12T19:25:48.521379Z","shell.execute_reply":"2023-03-12T19:26:36.109591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#model=build_model(best_hps) \n#model=get_model()\n","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:26:36.116024Z","iopub.execute_input":"2023-03-12T19:26:36.118311Z","iopub.status.idle":"2023-03-12T19:26:36.123264Z","shell.execute_reply.started":"2023-03-12T19:26:36.118263Z","shell.execute_reply":"2023-03-12T19:26:36.122094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#We train the model from scratch, please note that if you have trained enough epochs of the model during meshing, this step can be replaced by\n#model = tuner.get_best_models(num_models=1)[0]\n","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:26:36.125353Z","iopub.execute_input":"2023-03-12T19:26:36.125757Z","iopub.status.idle":"2023-03-12T19:26:36.134963Z","shell.execute_reply.started":"2023-03-12T19:26:36.125720Z","shell.execute_reply":"2023-03-12T19:26:36.133925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#DEFINE A MODEL FUNCTION AND HYPERPARAMETERS\nclass EpochPrintCB(tf.keras.callbacks.Callback):\n    def __init__(self, n_epochs_btwn_prints=5, extra_metrics_to_incl=None):\n        self.n_epochs_btwn_prints=n_epochs_btwn_prints        \n        self.extra_metrics_to_incl = extra_metrics_to_incl if ((extra_metrics_to_incl is None) or (type(extra_metrics_to_incl)==list)) else list(extra_metrics_to_incl)\n    \n    def on_epoch_end(self, epoch, logs):\n        if epoch % self.n_epochs_btwn_prints == 0:\n            print_str = f\"|| Epoch {epoch:>3} | lr: {self.model.optimizer.lr.numpy():10.7f} || loss:{logs['loss']:8.5f} | acc:{logs['acc']:8.5f} || val_loss:{logs['val_loss']:8.5f} | val_acc:{logs['val_acc']:8.5f} ||\"\n            if self.extra_metrics_to_incl is not None:\n                for extra_metric in self.extra_metrics_to_incl:\n                        print_str = \"||\".join([\n                            group if i in [0, 1, len(print_str.split(\"||\"))-1] else group[:-1]+f\" | {'val_' if group[1]=='v' else ''}{extra_metric}:{logs[('val_' if group[1]=='v' else '')+extra_metric]:8.5f} \"\n                            for i, group in enumerate(print_str.split(\"||\"))\n                        ])\n            print(print_str)\n\ndef fc_block(inputs, output_channels, dropout=0.2, _act=\"relu\"):\n    x = tf.keras.layers.Dense(output_channels)(inputs)\n    x = tf.keras.layers.BatchNormalization()(x)\n    x = tf.keras.layers.Activation(_act)(x)\n    x = tf.keras.layers.Dropout(dropout)(x)\n    return x\n\ndef get_model(n_labels=250, init_fc=512, n_blocks=2, _dropout_1=0.2, _dropout_2=0.6, _fc_step_rate=2, n_ax=2, n_feats=2,\n              types_to_use=(\"lips\", \"left_hand\", \"pose\", \"right_hand\"), do_L1=False, do_L2=False,\n              type_frame_len={\"lips\":43, \"left_hand\":21, \"pose\":33, \"right_hand\":21}):\n    \n    flat_frame_len = sum([type_frame_len[x]*n_ax*n_feats for x in types_to_use])\n    _inputs = tf.keras.layers.Input(shape=(flat_frame_len,))\n    x = _inputs\n    \n    # Define layers\n    for i in range(n_blocks):\n        x = fc_block(\n            x, output_channels=init_fc//(_fc_step_rate**i),\n            dropout=_dropout_1 if i!=(n_blocks-1) else _dropout_2\n        )\n    \n    # Define output layers\n    _outputs = tf.keras.layers.Dense(n_labels, activation=\"softmax\")(x)\n    \n    # Build the model\n    model = tf.keras.models.Model(inputs=_inputs, outputs=_outputs)\n    return model\n\nBATCH_SIZE   = 2048\nLR           = 0.0005\nDROPOUT_1    = 0.3\nDROPOUT_2    = 0.5\nN_EPOCHS     = 400\nINIT_FC      = 512\nN_BLOCKS     = 2\nFC_STEP_RATE = 1.25\nCB_MONITOR   = \"val_acc\"\nLOSS_FN      = \"sparse_categorical_crossentropy\"\nMETRICS      = [\"acc\", tf.keras.metrics.SparseTopKCategoricalAccuracy(k=5, name='t5_acc')]\n\nmodel_kwargs =dict(\n    init_fc=INIT_FC, \n    n_blocks=N_BLOCKS, \n    _dropout_1=DROPOUT_1, \n    _dropout_2=DROPOUT_2, \n    _fc_step_rate=FC_STEP_RATE,\n)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:26:36.136505Z","iopub.execute_input":"2023-03-12T19:26:36.137026Z","iopub.status.idle":"2023-03-12T19:26:36.158188Z","shell.execute_reply.started":"2023-03-12T19:26:36.136987Z","shell.execute_reply":"2023-03-12T19:26:36.157210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train_x,train_y\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:26:36.159925Z","iopub.execute_input":"2023-03-12T19:26:36.160408Z","iopub.status.idle":"2023-03-12T19:26:36.509868Z","shell.execute_reply.started":"2023-03-12T19:26:36.160366Z","shell.execute_reply":"2023-03-12T19:26:36.508667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold, StratifiedGroupKFold\nNP_FILE_DIR      = \"/kaggle/input/isolated-sign-language-aggregation-preparation\"\nall_x = np.load(os.path.join(NP_FILE_DIR, \"feature_data.npy\")).astype(np.float32)\nall_y = np.load(os.path.join(NP_FILE_DIR, \"feature_labels.npy\")).astype(np.uint8)\n\n# add nan back in not to screw up means/std\nall_x = np.where(all_x==0.0, np.nan, all_x)\n\n# Get mean and std ignoring nans\nall_mean = np.nanmean(all_x, keepdims=True, axis=0)\nall_std = np.nanstd(all_x, keepdims=True, axis=0)\n\n# Standardize around 0\nall_x = (all_x-all_mean)/all_std\n# Technically I don't need to do the were because np.nan \n# subtracting or dividing anything still results in np.nan\n#    - all_x = np.where(np.isnan(all_x), all_x, all_x-all_mean)\n#    - all_x = np.where(np.isnan(all_x), all_x, all_x/all_std)\n\n# Back to 0s\nall_x = np.nan_to_num(all_x)\n\n# There are 21 participants so we use 7 folds\n#    --> i.e. 3 participants in val every time\nN_PARTICIPANTS = train_df.participant_id.nunique()\n\n# Thanks Robert Hatch! \n#     --> https://www.kaggle.com/competitions/asl-signs/discussion/392335\n#\n# We are including 29302 in RH signer\nRH_SIGNERS = [26734, 28656, 25571, 62590, 29302, \n                       49445, 53618, 18796,  4718,  2044, \n                       37779, 30680]\n\n# We are including 37055 in LH Signer\nLH_SIGNERS  = [16069, 32319, 36257, 22343, 27610, 61333, 34503, 55372, 37055]\n\n# Since the sign label itself if relatively \n# There are 21 participants so we use 7 folds\n#    --> i.e. 3 participants in val every time\nK_FOLDS = 7\n\ndef get_folds(df, k_folds, force_one_lh=False, lh_signers=LH_SIGNERS[:-1]):\n    while True:\n        sgkf = StratifiedGroupKFold(n_splits=K_FOLDS, shuffle=True)\n        _fold_ds_idx_map = {\n            i:{\"train\":t_idxs, \"val\":v_idxs} \\\n            for i, (t_idxs, v_idxs) in enumerate(sgkf.split(df.index, df.sign, df.participant_id))\n        }\n        \n        # Ensure only one left hander in every val group\n        if force_one_lh:\n            if all([len(set(df.iloc[_idxs['val']][\"participant_id\"].unique()).intersection(set(lh_signers)))==1 for _idxs in _fold_ds_idx_map.values()]):\n                return _fold_ds_idx_map\n            else:\n                print(\".\", end=\"\")\n        else:\n            return _fold_ds_idx_map\n    \nfold_ds_idx_map = get_folds(train_df, K_FOLDS)\nprint(\" APPROPRIATE KFOLD SPLIT FOUND!\\n\")","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:26:36.511727Z","iopub.execute_input":"2023-03-12T19:26:36.513421Z","iopub.status.idle":"2023-03-12T19:26:41.385899Z","shell.execute_reply.started":"2023-03-12T19:26:36.513380Z","shell.execute_reply":"2023-03-12T19:26:41.384857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"histories, MODEL_DIR = [], \"/kaggle/working/models1\"\nif not os.path.isdir(MODEL_DIR): os.makedirs(MODEL_DIR)\n\nfor fold_num, fold_idxs in fold_ds_idx_map.items():    \n    print(f\"\\n\\n... STARTING TRAINING FOR FOLD #{fold_num+1} ...\\n\")\n    \n    # Get the dataset\n    val_x, val_y     = all_x[fold_idxs[\"val\"]],   all_y[fold_idxs[\"val\"]]\n    train_x, train_y = all_x[fold_idxs[\"train\"]], all_y[fold_idxs[\"train\"]]\n\n    # Initialize optimizer\n    optimizer = tf.keras.optimizers.Adam(LR)\n    \n    # Initialize CB list\n    _pct_to_drop = 3\n    cb_list = [\n        tf.keras.callbacks.EarlyStopping(patience=50, restore_best_weights=True, verbose=1, monitor=CB_MONITOR),\n        tf.keras.callbacks.ReduceLROnPlateau(patience=2, factor=(1-0.01*_pct_to_drop), verbose=0, monitor=CB_MONITOR),\n        EpochPrintCB(extra_metrics_to_incl=[\"t5_acc\",])\n    ]\n            \n    # Initialize model\n    model = get_model(**model_kwargs)\n    model.compile(optimizer, loss=LOSS_FN, metrics=METRICS)\n    \n    # See the structure and number of parameters\n    if fold_num==0: print(f\"\\n\\nFIRST FOLD... PRINTING MODEL SUMMARY:\\n\"); model.summary()\n        \n    # Fit!\n    print(\"\\n\\n... BEGINNING MODEL TRAINING ...\\n\")\n    if if_train:\n        histories.append(model.fit(train_x, train_y, validation_data=(val_x, val_y), epochs=N_EPOCHS, callbacks=cb_list, batch_size=BATCH_SIZE, verbose=0))\n    \n    else:\n        model = tf.keras.models.load_model(f'/kaggle/input/gislr-how-to-ensemble-version9/results/models/islr_model__fold_0{fold_num+1}')#The models are trained in version 14\n    \n    # Save\n    model.save(os.path.join(MODEL_DIR, f\"islr_model__fold_{fold_num+1:>02}__{model.evaluate(val_x, val_y, verbose=0)[1]:.5f}\").replace(\"0.\", \"\"))\n    \n    # Cleanup \n    del model, train_x, train_y, val_x, val_y; gc.collect(); gc.collect();","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:26:41.387843Z","iopub.execute_input":"2023-03-12T19:26:41.388971Z","iopub.status.idle":"2023-03-12T19:27:09.560175Z","shell.execute_reply.started":"2023-03-12T19:26:41.388910Z","shell.execute_reply":"2023-03-12T19:27:09.559071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#CREATE THE PREPROCESSING FUNCTION\ndef dumb_tf_mean(x, axis=None):\n    return tf.math.reduce_mean(x, axis=axis)\n\ndef dumb_tf_std(x, axis=None):\n    x = tf.experimental.numpy.var(x, axis=axis, dtype=tf.float32, ddof=1)\n    return tf.experimental.numpy.sqrt(x)\n\nclass PrepInputs(tf.keras.layers.Layer):\n    def __init__(self, lh_idx_range=(468, 489), pose_idx_range=(489, 522), rh_idx_range=(522, 543), distribution_mean=all_mean, distribution_std=all_std):\n        super(PrepInputs, self).__init__()\n        self.lips = tf.constant([61, 185, 40, 39, 37, 0, 267, 269, 270, 409, 291, 146, 91, 181, 84, 17, 314, 405, 321, 375, 291, 78, 191, 80, 81, 82, 13, 312, 311, 310, 415, 308, 78, 95, 88, 178, 87, 14, 317, 402, 318, 324, 308])\n        self.idx_ranges = [lh_idx_range, pose_idx_range, rh_idx_range]\n        self.flat_feat_lens = [2*self.lips.shape[0],]+[2*(_range[1]-_range[0]) for _range in self.idx_ranges]\n        self.distribution_mean = tf.constant(distribution_mean, dtype=tf.float32)\n        self.distribution_std  = tf.constant(distribution_std, dtype=tf.float32)\n    \n    def call(self, x_in):\n        \n        # Split the single vector into 4\n        xs = [tf.gather(x_in[..., :2], self.lips, axis=1),]+[x_in[:, _range[0]:_range[1], :2] for _range in self.idx_ranges]\n        \n        # Reshape based on specific number of keypoints\n        xs = [tf.reshape(_x, (-1, flat_feat_len)) for _x, flat_feat_len in zip(xs, self.flat_feat_lens)]\n        \n        # Drop empty rows - Empty rows are present in \n        #   --> face, lh, rh\n        #   --> so we don't have to for face\n        xs = [tf.boolean_mask(_x, tf.reduce_all(tf.logical_not(tf.math.is_nan(_x)), axis=1), axis=0) for _x in xs]\n        \n        # Get means and stds\n        x_means = [dumb_tf_mean(_x, axis=0) for _x in xs]\n        x_stds  = [dumb_tf_std(_x,  axis=0) for _x in xs]\n        \n        x_out = tf.concat([*x_means, *x_stds], axis=0)\n        x_out = tf.expand_dims(tf.where(tf.math.is_nan(x_out), tf.zeros_like(x_out), x_out), axis=0)\n        x_out = self.standardize_tensor(x_out)\n        return x_out\n    \n    def standardize_tensor(self, tensor):\n        return tf.where(tensor!=0, (tensor-self.distribution_mean)/self.distribution_std, tf.zeros_like(tensor))\n    \np_demo = PrepInputs()(load_relevant_data_subset(train_df.path[0]))\nprint(p_demo.shape)\ndef dumb_tf_mean(x, axis=None):\n    return tf.math.reduce_mean(x, axis=axis)\n\ndef dumb_tf_std(x, axis=None):\n    x = tf.experimental.numpy.var(x, axis=axis, dtype=tf.float32, ddof=1)\n    return tf.experimental.numpy.sqrt(x)\n\nclass PrepInputs(tf.keras.layers.Layer):\n    def __init__(self, lh_idx_range=(468, 489), pose_idx_range=(489, 522), rh_idx_range=(522, 543), distribution_mean=all_mean, distribution_std=all_std):\n        super(PrepInputs, self).__init__()\n        self.lips = tf.constant([61, 185, 40, 39, 37, 0, 267, 269, 270, 409, 291, 146, 91, 181, 84, 17, 314, 405, 321, 375, 291, 78, 191, 80, 81, 82, 13, 312, 311, 310, 415, 308, 78, 95, 88, 178, 87, 14, 317, 402, 318, 324, 308])\n        self.idx_ranges = [lh_idx_range, pose_idx_range, rh_idx_range]\n        self.flat_feat_lens = [2*self.lips.shape[0],]+[2*(_range[1]-_range[0]) for _range in self.idx_ranges]\n        self.distribution_mean = tf.constant(distribution_mean, dtype=tf.float32)\n        self.distribution_std  = tf.constant(distribution_std, dtype=tf.float32)\n    \n    def call(self, x_in):\n        \n        # Split the single vector into 4\n        xs = [tf.gather(x_in[..., :2], self.lips, axis=1),]+[x_in[:, _range[0]:_range[1], :2] for _range in self.idx_ranges]\n        \n        # Reshape based on specific number of keypoints\n        xs = [tf.reshape(_x, (-1, flat_feat_len)) for _x, flat_feat_len in zip(xs, self.flat_feat_lens)]\n        \n        # Drop empty rows - Empty rows are present in \n        #   --> face, lh, rh\n        #   --> so we don't have to for face\n        xs = [tf.boolean_mask(_x, tf.reduce_all(tf.logical_not(tf.math.is_nan(_x)), axis=1), axis=0) for _x in xs]\n        \n        # Get means and stds\n        x_means = [dumb_tf_mean(_x, axis=0) for _x in xs]\n        x_stds  = [dumb_tf_std(_x,  axis=0) for _x in xs]\n        \n        x_out = tf.concat([*x_means, *x_stds], axis=0)\n        x_out = tf.expand_dims(tf.where(tf.math.is_nan(x_out), tf.zeros_like(x_out), x_out), axis=0)\n        x_out = self.standardize_tensor(x_out)\n        return x_out\n    \n    def standardize_tensor(self, tensor):\n        return tf.where(tensor!=0, (tensor-self.distribution_mean)/self.distribution_std, tf.zeros_like(tensor))\n    \np_demo = PrepInputs()(load_relevant_data_subset(train_df.path[0]))\nprint(p_demo.shape)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:27:09.563292Z","iopub.execute_input":"2023-03-12T19:27:09.563701Z","iopub.status.idle":"2023-03-12T19:27:09.726743Z","shell.execute_reply.started":"2023-03-12T19:27:09.563659Z","shell.execute_reply":"2023-03-12T19:27:09.725721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MODEL_PATHS = sorted(glob(os.path.join(MODEL_DIR, \"*\")), key=lambda x: int(x.rsplit(\"__\", 1)[-1]), reverse=True)\n\nclass ISLRModel(tf.keras.Model):\n    \"\"\"\n    TensorFlow Lite model that takes input tensors and applies:\n        – a preprocessing model\n        – the ISLR model \n    \"\"\"\n\n    def __init__(self, islr_fold_models):\n        \"\"\"\n        Initializes the TFLiteModel with the specified preprocessing model and ISLR model.\n        \"\"\"\n        super(ISLRModel, self).__init__()\n\n        # Load the feature generation and main models\n        self.islr_fold_models  = list(islr_fold_models.values())\n        self.model_weights = tf.repeat(tf.expand_dims(tf.constant([float(k)/100_000. for k in islr_fold_models.keys()], dtype=tf.float32), axis=-1), 250, axis=-1)\n    \n    def __call__(self, inputs, training=None):\n        \"\"\"\n        Applies the feature generation model and main model to the input tensors.\n\n        Args:\n            inputs: Input tensor with shape [batch_size, 543, 3].\n\n        Returns:\n            A dictionary with a single key 'outputs' and corresponding output tensor.\n        \"\"\"\n        batch_size = tf.shape(inputs)[0]\n        outputs    = tf.concat([_model(inputs, training=training) for _model in self.islr_fold_models], axis=0)\n        outputs = tf.reduce_mean(outputs, axis=0, keepdims=True)\n\n        ### Compute the weighted sum and the sum of the weights and compute the weighted mean\n        #outputs    = tf.stack([_model(inputs, training=training) for _model in self.islr_fold_models], axis=1)\n        #outputs = tf.reduce_sum(tf.multiply(outputs, tf.repeat(tf.expand_dims(self.model_weights, axis=0), batch_size, axis=0)), axis=1)\n        #outputs = tf.divide(outputs, tf.reduce_sum(tf.repeat(tf.expand_dims(self.model_weights, axis=0), batch_size, axis=0), axis=1))\n        \n        # Return a dictionary with the output tensor\n        return outputs\n\nclass TFLiteModel(tf.Module):\n    \"\"\"\n    TensorFlow Lite model that takes input tensors and applies:\n        – a preprocessing model\n        – the ISLR model \n    \"\"\"\n\n    def __init__(self, islr_fold_models, islr_fold_pp_fn):\n        \"\"\"\n        Initializes the TFLiteModel with the specified preprocessing model and ISLR model.\n        \"\"\"\n        super(TFLiteModel, self).__init__()\n\n        # Load the feature generation and main models\n        self.prep_inputs = islr_fold_pp_fn()\n        self.islr_fold_models  = list(islr_fold_models.values())\n        self.model_weights = tf.repeat(tf.expand_dims(tf.constant([float(k)/100_000. for k in islr_fold_models.keys()], dtype=tf.float32), axis=-1), 250, axis=-1)\n    \n    @tf.function(input_signature=[tf.TensorSpec(shape=[None, 543, 3], dtype=tf.float32, name='inputs')])\n    def __call__(self, inputs):\n        \"\"\"\n        Applies the feature generation model and main model to the input tensors.\n\n        Args:\n            inputs: Input tensor with shape [batch_size, 543, 3].\n\n        Returns:\n            A dictionary with a single key 'outputs' and corresponding output tensor.\n        \"\"\"\n        x = self.prep_inputs(tf.cast(inputs, dtype=tf.float32))\n        outputs  = tf.concat([_model(x) for _model in self.islr_fold_models], axis=0)\n        \n        # Compute the weighted sum and the sum of the weights and compute the weighted mean\n        outputs = tf.reduce_sum(tf.multiply(outputs, self.model_weights), axis=0)\n        outputs = tf.divide(outputs, tf.reduce_sum(self.model_weights, axis=0))\n        \n        # Return a dictionary with the output tensor\n        return {'outputs': outputs}\n\nISLR_FOLD_MODELS = {_path.rsplit(\"__\", 1)[-1]:tf.keras.models.load_model(_path, compile=False) for _path in MODEL_PATHS}\ntflite_keras_model = TFLiteModel(ISLR_FOLD_MODELS, PrepInputs)\nout = tflite_keras_model(load_relevant_data_subset(train_df.path[0]))[\"outputs\"]\nnp.argmax(out)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:27:09.728162Z","iopub.execute_input":"2023-03-12T19:27:09.728534Z","iopub.status.idle":"2023-03-12T19:27:14.122308Z","shell.execute_reply.started":"2023-03-12T19:27:09.728494Z","shell.execute_reply":"2023-03-12T19:27:14.121303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_input_shape(num_frames, landmarks, flag_drop_z):\n    input_shape = (num_frames, landmarks * 3)\n\n    if flag_drop_z:\n        num_coords = 2\n    else:\n        num_coords = 3\n\n    return (num_frames, landmarks * num_coords)\n\noutput_bias = tf.keras.initializers.Constant(1.0 / 250.0)\nclass MSD(tf.keras.layers.Layer):\n    def __init__(\n        self,\n        units,\n        fold_num=1,\n        **kwargs,\n    ):\n        super().__init__(**kwargs)\n\n        self.lin = tf.keras.layers.Dense(\n            units,\n            activation=None,\n            use_bias=True,\n            bias_initializer=output_bias,\n            # kernel_regularizer=R.l2(WEIGHT_REGULARIZE)\n        )\n\n        rate_dropout = 0.5\n        self.dropouts = [\n            tf.keras.layers.Dropout((rate_dropout - 0.2), seed=135 + fold_num),\n            tf.keras.layers.Dropout((rate_dropout - 0.1), seed=690 + fold_num),\n            tf.keras.layers.Dropout((rate_dropout), seed=275 + fold_num),\n            tf.keras.layers.Dropout((rate_dropout + 0.1), seed=348 + fold_num),\n            tf.keras.layers.Dropout((rate_dropout + 0.2), seed=861 + fold_num),\n        ]\n\n    def call(self, inputs):\n        for ii, drop in enumerate(self.dropouts):\n            if ii == 0:\n                out = self.lin(drop(inputs)) / 5.0\n            else:\n                out += self.lin(drop(inputs)) / 5.0\n        return out\n\n\nclass ResidualBlock(tf.keras.layers.Layer):\n    def __init__(self, units, dropout):\n        super().__init__()\n        self.linear = tf.keras.layers.Dense(units)\n        self.bn = tf.keras.layers.BatchNormalization()\n        self.act = tf.keras.layers.Activation(\"gelu\")\n        if dropout != 0:\n            self.drop = tf.keras.layers.Dropout(dropout)\n            self.flag_use_drop = True\n        else:\n            self.flag_use_drop = False\n\n    def call(self, x):\n        x = self.linear(x)\n        x = self.bn(x)\n        x = self.act(x)\n        if self.flag_use_drop:\n            x = self.drop(x)\n        return x\n\nclass GRUModel(tf.keras.layers.Layer):\n    def __init__(self, units, dropout, num_blocks):\n        super().__init__()\n        self.start_gru = tf.keras.layers.GRU(\n            units=units, dropout=0.0, return_sequences=True\n        )\n        self.end_gru = tf.keras.layers.GRU(\n            units=units, dropout=dropout, return_sequences=False\n        )\n\n        if (num_blocks - 2) > 0:\n            self.gru_blocks = [\n                tf.keras.layers.GRU(units=units, dropout=dropout, return_sequences=True)\n                * (num_blocks - 2)\n            ]\n            self.flag_use_gru_blocks = True\n        else:\n            self.flag_use_gru_blocks = False\n\n    def call(self, x):\n        x = self.start_gru(x)\n        if self.flag_use_gru_blocks:\n            for blk in self.gru_blocks:\n                x = blk(x)\n        x = self.end_gru(x)\n        return x\n\ndef model_utils(cfg, fold_num):\n    metric_ls = [\n        tf.keras.metrics.SparseCategoricalAccuracy(),\n        tf.keras.metrics.SparseTopKCategoricalAccuracy(k=5),\n    ]\n\n    cb_list = [\n        tf.keras.callbacks.EarlyStopping(\n            patience=5,\n            restore_best_weights=True,\n            verbose=1,\n            monitor=cfg[\"TARGET_METRIC\"],\n        ),\n        tf.keras.callbacks.ReduceLROnPlateau(patience=2, factor=0.8, verbose=1),\n        tf.keras.callbacks.ModelCheckpoint(\n            f\"{SAVE_DIR}/best_acc_{fold_num}.h5\",\n            monitor=cfg[\"TARGET_METRIC\"],\n            verbose=0,\n            save_best_only=True,\n            save_weights_only=True,\n            mode=\"max\",\n            save_freq=\"epoch\",\n        ),\n    ]\n\n    if cfg[\"FLAG_WANDB\"]:\n        cb_list += [#WandbMetricsLogger()\n            WandbCallback(\n                monitor=cfg[\"TARGET_METRIC\"],\n                log_weights=False,\n                log_evaluation=False,\n                save_model=False,\n            )\n        ]\n\n    opt = tfa.optimizers.AdamW(weight_decay=0, learning_rate=cfg[\"LR\"])\n    # opt = tf.keras.optimizers.Adam(learning_rate=LR)\n    # opt = tfa.optimizers.RectifiedAdam(learning_rate=LR)\n    # opt = tfa.optimizers.Lookahead(opt, sync_period=5)\n\n    return metric_ls, cb_list, opt\n\n# Analyzing Handedness\nleft_handed_signer = [16069, 32319, 36257, 22343, 27610, 61333, 34503, 55372, 37055]  # both_hands_signer-> 37055\nright_handed_signer = [26734, 28656, 25571, 62590, 29302, 49445, 53618, 18796, 4718, 2044, 37779, 30680,]\nlip_landmarks = [61, 185, 40, 39, 37, 0, 267, 269, 270, 409, 291, 146, 91, 181, 84, 17, 314, 405, 321, 375, 78, 191, 80, 81, 82, 13, 312, 311, 310, 415, 95, 88, 178, 87, 14, 317, 402, 318, 324, 308]\n\ndi = {}\nfor k in left_handed_signer:\n    di[k] = 0\nfor k in right_handed_signer:\n    di[k] = 1\n\nleft_hand_landmarks = list(range(468, 468 + 21))\nright_hand_landmarks = list(range(522, 522 + 21))\n\naveraging_sets = [\n    [0, 468],\n    [489, 33],\n]  ## average over the entire face, and the entire 'pose'\n\npoint_landmarks = [\n    item\n    for sublist in [lip_landmarks, left_hand_landmarks, right_hand_landmarks]\n    for item in sublist\n]\n\nLANDMARKS = len(point_landmarks) #+ len(averaging_sets)\n\n# Fixed  ##################################################################################\n\nFLAG_DROP_Z = False\nROWS_PER_FRAME = 543\nNUM_FRAMES = 15\nINPUT_SHAPE = get_input_shape(NUM_FRAMES, LANDMARKS, FLAG_DROP_Z)\nSEGMENTS = 3\nNUM_BASE_FEATS = (SEGMENTS + 1) * INPUT_SHAPE[1] * 2\nFLAT_FRAME_SHAPE = NUM_BASE_FEATS + (INPUT_SHAPE[0] * INPUT_SHAPE[1])\ndecoder = {v: k for k, v in read_json_file(\"/kaggle/input/asl-signs/sign_to_prediction_index_map.json\").items()}\n\n    \n_inputs = tf.keras.layers.Input(shape=(FLAT_FRAME_SHAPE,))\n\n# import ipdb\n# ipdb.set_trace()\nx = _inputs[:, :NUM_BASE_FEATS]\nx_conv = tf.reshape(_inputs[:, NUM_BASE_FEATS:], (-1, NUM_FRAMES, INPUT_SHAPE[1]))\n\n# Concat Dilated Convolutions with actual data\ngru_out = GRUModel(512, 0.5, 1)(x_conv)\nx = gru_out\n\n# Residual Block\nx = ResidualBlock(1024, 0.25)(x)\nx += ResidualBlock(1024, 0.0)(x)\n\n# Final output MSD Layer\nx = MSD(units=250)(x)\n_outputs = tf.keras.layers.Softmax(dtype=\"float32\")(x)\n\n# Build the model\ngwg_models=[]\nfor i in range(5):\n    gwg_model = tf.keras.models.Model(inputs=_inputs, outputs=_outputs)\n    gwg_model.summary()\n    gwg_model.load_weights(f\"/kaggle/input/islr-external-3-my/models/best_acc_{i}.h5\")\n    gwg_models.append(gwg_model)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:27:14.123776Z","iopub.execute_input":"2023-03-12T19:27:14.124151Z","iopub.status.idle":"2023-03-12T19:27:17.334075Z","shell.execute_reply.started":"2023-03-12T19:27:14.124112Z","shell.execute_reply":"2023-03-12T19:27:17.333022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def tf_nan_mean(x, axis=0):\n    return tf.reduce_sum(tf.where(tf.math.is_nan(x), tf.zeros_like(x), x), axis=axis) / tf.reduce_sum(tf.where(tf.math.is_nan(x), tf.zeros_like(x), tf.ones_like(x)), axis=axis)\n\ndef tf_nan_std(x, axis=0):\n    d = x - tf_nan_mean(x, axis=axis)\n    return tf.math.sqrt(tf_nan_mean(d * d, axis=axis))\n\ndef flatten_means_and_stds(x, axis=0):\n    # Get means and stds\n    x_mean = tf_nan_mean(x, axis=0)\n    x_std  = tf_nan_std(x,  axis=0)\n\n    x_out = tf.concat([x_mean, x_std], axis=0)\n    x_out = tf.reshape(x_out, (1, INPUT_SHAPE[1]*2))\n    x_out = tf.where(tf.math.is_finite(x_out), x_out, tf.zeros_like(x_out))\n    return x_out\n\nclass FeatureGen_1(tf.keras.layers.Layer):\n    def __init__(self):\n        super(FeatureGen_1, self).__init__()\n    \n    def call(self, x_in):\n        x = tf.gather(x_in, point_landmarks, axis=1)\n\n        x_padded = x\n        for i in range(SEGMENTS):\n            p0 = tf.where( ((tf.shape(x_padded)[0] % SEGMENTS) > 0) & ((i % 2) != 0) , 1, 0)\n            p1 = tf.where( ((tf.shape(x_padded)[0] % SEGMENTS) > 0) & ((i % 2) == 0) , 1, 0)\n            paddings = [[p0, p1], [0, 0], [0, 0]]\n            x_padded = tf.pad(x_padded, paddings, mode=\"SYMMETRIC\")\n        x_list = tf.split(x_padded, SEGMENTS)\n        x_list = [flatten_means_and_stds(_x, axis=0) for _x in x_list]\n\n        x_list.append(flatten_means_and_stds(x, axis=0))\n        \n        ## Resize only dimension 0. Resize can't handle nan, so replace nan with that dimension's avg value to reduce impact.\n        x = tf.image.resize(tf.where(tf.math.is_finite(x), x, tf_nan_mean(x, axis=0)), [NUM_FRAMES, LANDMARKS])\n        x = tf.reshape(x, (1, INPUT_SHAPE[0]*INPUT_SHAPE[1]))\n        x = tf.where(tf.math.is_nan(x), tf.zeros_like(x), x)\n        x_list.append(x)\n        x = tf.concat(x_list, axis=1)\n        return x","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:27:17.335974Z","iopub.execute_input":"2023-03-12T19:27:17.336724Z","iopub.status.idle":"2023-03-12T19:27:17.351124Z","shell.execute_reply.started":"2023-03-12T19:27:17.336683Z","shell.execute_reply":"2023-03-12T19:27:17.349993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class ISLRModel(tf.keras.Model):\n    \"\"\"\n    TensorFlow Lite model that takes input tensors and applies:\n        – a preprocessing model\n        – the ISLR model \n    \"\"\"\n\n    def __init__(self, islr_fold_models):\n        \"\"\"\n        Initializes the TFLiteModel with the specified preprocessing model and ISLR model.\n        \"\"\"\n        super(ISLRModel, self).__init__()\n\n        # Load the feature generation and main models\n        self.islr_fold_models  = list(islr_fold_models.values())\n        self.model_weights = tf.repeat(tf.expand_dims(tf.constant([float(k)/100_000. for k in islr_fold_models.keys()], dtype=tf.float32), axis=-1), 250, axis=-1)\n    \n    def __call__(self, inputs, training=None):\n        \"\"\"\n        Applies the feature generation model and main model to the input tensors.\n\n        Args:\n            inputs: Input tensor with shape [batch_size, 543, 3].\n\n        Returns:\n            A dictionary with a single key 'outputs' and corresponding output tensor.\n        \"\"\"\n        batch_size = tf.shape(inputs)[0]\n        outputs    = tf.concat([_model(inputs, training=training) for _model in self.islr_fold_models], axis=0)\n        outputs = tf.reduce_mean(outputs, axis=0, keepdims=True)\n\n        ### Compute the weighted sum and the sum of the weights and compute the weighted mean\n        #outputs    = tf.stack([_model(inputs, training=training) for _model in self.islr_fold_models], axis=1)\n        #outputs = tf.reduce_sum(tf.multiply(outputs, tf.repeat(tf.expand_dims(self.model_weights, axis=0), batch_size, axis=0)), axis=1)\n        #outputs = tf.divide(outputs, tf.reduce_sum(tf.repeat(tf.expand_dims(self.model_weights, axis=0), batch_size, axis=0), axis=1))\n        \n        # Return a dictionary with the output tensor\n        return outputs\n\nclass TFLiteModel(tf.Module):\n    \"\"\"\n    TensorFlow Lite model that takes input tensors and applies:\n        – a preprocessing model\n        – the ISLR model \n    \"\"\"\n\n    def __init__(self, islr_fold_models, islr_fold_pp_fn):\n        \"\"\"\n        Initializes the TFLiteModel with the specified preprocessing model and ISLR model.\n        \"\"\"\n        super(TFLiteModel, self).__init__()\n\n        # Load the feature generation and main models\n        self.prep_inputs = islr_fold_pp_fn()\n        self.islr_fold_models  = list(islr_fold_models.values())\n        self.model_weights = tf.repeat(tf.expand_dims(tf.constant([float(k)/100_000. for k in islr_fold_models.keys()], dtype=tf.float32), axis=-1), 250, axis=-1)\n    \n    @tf.function(input_signature=[tf.TensorSpec(shape=[None, 543, 3], dtype=tf.float32, name='inputs')])\n    def __call__(self, inputs):\n        \"\"\"\n        Applies the feature generation model and main model to the input tensors.\n\n        Args:\n            inputs: Input tensor with shape [batch_size, 543, 3].\n\n        Returns:\n            A dictionary with a single key 'outputs' and corresponding output tensor.\n        \"\"\"\n        x = self.prep_inputs(tf.cast(inputs, dtype=tf.float32))\n        outputs  = tf.concat([_model(x) for _model in self.islr_fold_models], axis=0)\n        \n        # Compute the weighted sum and the sum of the weights and compute the weighted mean\n        outputs = tf.reduce_sum(tf.multiply(outputs, self.model_weights), axis=0)\n        outputs = tf.divide(outputs, tf.reduce_sum(self.model_weights, axis=0))\n        \n        # Return a dictionary with the output tensor\n        return {'outputs': outputs}\n\n#ISLR_FOLD_MODELS = {_path.rsplit(\"__\", 1)[-1]:tf.keras.models.load_model(_path, compile=False) for _path in MODEL_PATHS}\n#tflite_keras_model = TFLiteModel(ISLR_FOLD_MODELS, PrepInputs)\n#out = tflite_keras_model(load_relevant_data_subset(train_df.path[0]))[\"outputs\"]\n#np.argmax(out)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:27:17.352882Z","iopub.execute_input":"2023-03-12T19:27:17.353876Z","iopub.status.idle":"2023-03-12T19:27:17.369189Z","shell.execute_reply.started":"2023-03-12T19:27:17.353837Z","shell.execute_reply":"2023-03-12T19:27:17.368204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class TFLiteModel(tf.Module):\n    \"\"\"\n    TensorFlow Lite model that takes input tensors and applies:\n        – a preprocessing model\n        – the ISLR model \n    \"\"\"\n\n    def __init__(self, islr_fold_models, islr_fold_pp_fn, gwg_model, gwg_pp_fn):\n        \"\"\"\n        Initializes the TFLiteModel with the specified preprocessing model and ISLR model.\n        \"\"\"\n        super(TFLiteModel, self).__init__()\n\n        # Load the feature generation and main models\n        self.prep_inputs_1 = islr_fold_pp_fn()\n        self.prep_inputs_2 = gwg_pp_fn()\n        self.models_1      = list(islr_fold_models.values())\n        self.models_2       = gwg_models\n    \n    @tf.function(input_signature=[tf.TensorSpec(shape=[None, 543, 3], dtype=tf.float32, name='inputs')])\n    def __call__(self, inputs):\n        \"\"\"\n        Applies the feature generation model and main model to the input tensors.\n\n        Args:\n            inputs: Input tensor with shape [batch_size, 543, 3].\n\n        Returns:\n            A dictionary with a single key 'outputs' and corresponding output tensor.\n        \"\"\"\n        x1 = self.prep_inputs_1(tf.cast(inputs, dtype=tf.float32))\n        x2 = self.prep_inputs_2(tf.cast(inputs, dtype=tf.float32))\n        \n        outputs_1 = tf.keras.layers.Average()([_model(x1) for _model in self.models_1])\n        \n        outputs_2=[]\n        for gwg_model in self.models_2:\n            outputs_2.append(gwg_model(x2))\n        outputs_2 = tf.keras.layers.Average()(outputs_2)\n        \n        # 2x weighting higher score via repeat 14\n        outputs = tf.multiply(0.4, outputs_1)+tf.multiply(0.6, outputs_2)\n        \n        # Return a dictionary with the output tensor\n        return {'outputs': outputs}\n    \nISLR_FOLD_MODELS = {_path.rsplit(\"__\", 1)[-1]:tf.keras.models.load_model(_path, compile=False) for _path in MODEL_PATHS}\ntflite_keras_model = TFLiteModel(ISLR_FOLD_MODELS, PrepInputs, gwg_models, FeatureGen_1)\nout = tflite_keras_model(load_relevant_data_subset(train_df.path[0]))[\"outputs\"]\nnp.argmax(out)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:27:17.375451Z","iopub.execute_input":"2023-03-12T19:27:17.375733Z","iopub.status.idle":"2023-03-12T19:27:26.810921Z","shell.execute_reply.started":"2023-03-12T19:27:17.375706Z","shell.execute_reply":"2023-03-12T19:27:26.809857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generate TFLite Model and submission.zip","metadata":{}},{"cell_type":"code","source":"class TFLiteModel(tf.Module):\n    \"\"\"\n    TensorFlow Lite model that takes input tensors and applies:\n        – a preprocessing model\n        – the ASL model \n    \"\"\"\n\n    def __init__(self, asl_models):\n        \"\"\"\n        Initializes the TFLiteModel with the specified feature generation model and main model.\n        \"\"\"\n        super(TFLiteModel, self).__init__()\n\n        # Load the feature generation and main models\n        self.prep_inputs = FeatureGen()\n        self.asl_models   = asl_models\n    \n    @tf.function(input_signature=[tf.TensorSpec(shape=[None, 543, 3], dtype=tf.float32, name='inputs')])\n    def __call__(self, inputs):\n        \"\"\"\n        Applies the feature generation model and main model to the input tensors.\n\n        Args:\n            inputs: Input tensor with shape [batch_size, 543, 3].\n\n        Returns:\n            A dictionary with a single key 'outputs' and corresponding output tensor.\n        \"\"\"\n        x = self.prep_inputs(tf.cast(inputs, dtype=tf.float32))\n        outputs=[]\n        for asl_model in self.asl_models:\n            outputs.append(asl_model(x)[0, :])\n        outputs = tf.keras.layers.Average()(outputs)\n        #print(outputs)\n        # Return a dictionary with the output tensor\n        return {'outputs': outputs}\n\n#tflite_keras_model = TFLiteModel(models[:4])#Due to model size limitations, I only take the 4 with the best CV\n#demo_output = tflite_keras_model(load_relevant_data_subset(train_df.path[0]))[\"outputs\"]\n#decoder(np.argmax(demo_output.numpy(), axis=-1))","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:27:26.815276Z","iopub.execute_input":"2023-03-12T19:27:26.818130Z","iopub.status.idle":"2023-03-12T19:27:26.831010Z","shell.execute_reply.started":"2023-03-12T19:27:26.818089Z","shell.execute_reply":"2023-03-12T19:27:26.830010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class TFLiteModel(tf.Module):\n    \"\"\"\n    TensorFlow Lite model that takes input tensors and applies:\n        – a preprocessing model\n        – the ISLR model \n    \"\"\"\n\n    def __init__(self, islr_fold_models, islr_fold_pp_fn, gwg_model, gwg_pp_fn,):\n        \"\"\"\n        Initializes the TFLiteModel with the specified preprocessing model and ISLR model.\n        \"\"\"\n        super(TFLiteModel, self).__init__()\n\n        # Load the feature generation and main models\n \n        self.prep_inputs_1 = islr_fold_pp_fn()\n        self.prep_inputs_2 = gwg_pp_fn()\n        self.models_1      = list(islr_fold_models.values())\n        self.model_2       = gwg_model\n    \n    @tf.function(input_signature=[tf.TensorSpec(shape=[None, 543, 3], dtype=tf.float32, name='inputs')])\n    def __call__(self, inputs):\n        \"\"\"\n        Applies the feature generation model and main model to the input tensors.\n\n        Args:\n            inputs: Input tensor with shape [batch_size, 543, 3].\n\n        Returns:\n            A dictionary with a single key 'outputs' and corresponding output tensor.\n        \"\"\"\n        \n        \n        \n        x1 = self.prep_inputs_1(tf.cast(inputs, dtype=tf.float32))\n        x2 = self.prep_inputs_2(tf.cast(inputs, dtype=tf.float32))\n        \n        outputs_1 = tf.keras.layers.Average()([_model(x1) for _model in self.models_1])\n        outputs_2 = gwg_model(x2) \n        #print(outputs_1)\n        #print('----------------')\n        #print(outputs_2)\n        # 2x weighting higher score via repeat 14\n        \n        #outputs = tf.reduce_mean(tf.concat([outputs_1, tf.repeat(outputs_2, 14, axis=0)], axis=0), axis=0, keepdims=True)\n        \n        outputs = tf.multiply(0.3, outputs_1)+tf.multiply(0.7, outputs_2)\n        #print(outputs)\n        # Return a dictionary with the output tensor\n        return {'outputs': outputs}\n    \n#ISLR_FOLD_MODELS = {_path.rsplit(\"__\", 1)[-1]:tf.keras.models.load_model(_path, compile=False) for _path in MODEL_PATHS}\n#tflite_keras_model = TFLiteModel(ISLR_FOLD_MODELS, PrepInputs, gwg_model, FeatureGen_1)\n#out = tflite_keras_model(load_relevant_data_subset(train_df.path[0]))[\"outputs\"]\n#np.argmax(out)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:27:26.836081Z","iopub.execute_input":"2023-03-12T19:27:26.836857Z","iopub.status.idle":"2023-03-12T19:27:26.850267Z","shell.execute_reply.started":"2023-03-12T19:27:26.836818Z","shell.execute_reply":"2023-03-12T19:27:26.849161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install tflite-runtime\nimport tflite_runtime.interpreter as tflite","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:27:26.855380Z","iopub.execute_input":"2023-03-12T19:27:26.858373Z","iopub.status.idle":"2023-03-12T19:27:37.175003Z","shell.execute_reply.started":"2023-03-12T19:27:26.858333Z","shell.execute_reply":"2023-03-12T19:27:37.173435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"keras_model_converter = tf.lite.TFLiteConverter.from_keras_model(tflite_keras_model)\n#keras_model_converter.optimizations = [tf.lite.Optimize.DEFAULT]\ntflite_model = keras_model_converter.convert()\nwith open('/kaggle/working/models/model.tflite', 'wb') as f:\n    f.write(tflite_model)\n!zip submission.zip /kaggle/working/models/model.tflite\n\ninterpreter = tflite.Interpreter(\"/kaggle/working/models/model.tflite\")\nfound_signatures = list(interpreter.get_signature_list().keys())\n# if REQUIRED_SIGNATURE not in found_signatures:\n#     raise KernelEvalException('Required input signature not found.')\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\n\noutput = prediction_fn(inputs=load_relevant_data_subset(train_df.path[0]))\nsign = np.argmax(output[\"outputs\"])\n\nprint(\"PRED : \", decoder[sign])\nprint(\"GT   : \", train_df.sign[0])","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:27:37.177027Z","iopub.execute_input":"2023-03-12T19:27:37.177810Z","iopub.status.idle":"2023-03-12T19:28:35.326872Z","shell.execute_reply.started":"2023-03-12T19:27:37.177761Z","shell.execute_reply":"2023-03-12T19:28:35.325370Z"},"trusted":true},"execution_count":null,"outputs":[]}]}