\n ![\"Google](\"https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png\")
Isolated Sign Language Recognition
\n \n ![\"asl](\"https://cdn-cbkob.nitrocdn.com/TiGMibPGMREAJjbFYNLfxxdkUjUGroSw/assets/images/optimized/rev-22fc791/wp-content/uploads/2022/12/sign-language.jpg\")
CREATED BY: SARASWATI TIWARI
","metadata":{}},{"cell_type":"markdown","source":"**The goal of this competition is to classify isolated American Sign Language (ASL) signs. I will create a TensorFlow Lite model trained on labeled landmark data extracted using the MediaPipe Holistic Solution.**\n\n**My work may improve the ability of PopSign to help relatives of deaf children learn basic signs and communicate better with their loved ones.**","metadata":{}},{"cell_type":"markdown","source":"**Why TensorFlow Lite**\n\nTo allow the ML model to run on device in an attempt to limit latency inside the game, PopSign doesn’t send user videos to the cloud. Therefore, all inference must be done on the phone itself. PopSign is building its recognition pipeline on top of TensorFlow Lite, which runs on both Android and iOS.","metadata":{}},{"cell_type":"markdown","source":"**PopSign** *is an app developed by the Georgia Institute of Technology and the National Technical Institute for the Deaf at Rochester Institute of Technology. The app is available in beta on Android and iOS.*","metadata":{}},{"cell_type":"code","source":"print(\"\\n... PIP INSTALLS STARTING ...\\n\")\nprint(\"\\n... PIP INSTALLS COMPLETE ...\\n\")\n\n\nprint(\"\\n... IMPORTS STARTING ...\\n\")\nprint(\"\\n\\tVERSION INFORMATION\")\n# Competition Specific Imports (You'll see why we need these later)\n# mediapipe above\n\n# Machine Learning and Data Science Imports (basics)\nimport tensorflow as tf; print(f\"\\t\\t– TENSORFLOW VERSION: {tf.__version__}\");\n# import tensorflow_io as tfio; print(f\"\\t\\t– TENSORFLOW-IO VERSION: {tfio.__version__}\");\nimport pandas as pd; pd.options.mode.chained_assignment = None; pd.set_option('display.max_columns', None);\nimport numpy as np; print(f\"\\t\\t– NUMPY VERSION: {np.__version__}\");\nimport sklearn; print(f\"\\t\\t– SKLEARN VERSION: {sklearn.__version__}\");\n\n# Built-In Imports (mostly don't worry about these)\nfrom kaggle_datasets import KaggleDatasets\nfrom collections import Counter\nfrom datetime import datetime\nfrom zipfile import ZipFile\nfrom glob import glob\nimport Levenshtein\nimport warnings\nimport requests\nimport hashlib\nimport imageio\nimport IPython\nimport sklearn\nimport urllib\nimport zipfile\nimport pickle\nimport random\nimport shutil\nimport string\nimport json\nimport math\nimport time\nimport gzip\nimport ast\nimport sys\nimport io\nimport os\nimport gc\nimport re\n\nprint(\"\\n\\n... IMPORTS COMPLETE ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2023-03-14T22:46:07.676417Z","iopub.execute_input":"2023-03-14T22:46:07.677213Z","iopub.status.idle":"2023-03-14T22:46:16.319099Z","shell.execute_reply.started":"2023-03-14T22:46:07.677153Z","shell.execute_reply":"2023-03-14T22:46:16.317885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"models = []\nweights = []","metadata":{"execution":{"iopub.status.busy":"2023-03-14T22:46:16.321520Z","iopub.execute_input":"2023-03-14T22:46:16.322302Z","iopub.status.idle":"2023-03-14T22:46:16.327256Z","shell.execute_reply.started":"2023-03-14T22:46:16.322263Z","shell.execute_reply":"2023-03-14T22:46:16.326181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"models.append(tf.keras.models.load_model(\"/kaggle/input/gislr-lb-0-63-on-the-shoulders/models/final_model\"))\nweights.append(0.8)\n\nprint(models[-1](tf.keras.Input(shape=(543, 3))))\n\nmodels.append(tf.keras.models.load_model(\"/kaggle/input/gislr-small-version-on-the-shoulders/models/final_model\"))\nweights.append(0.4)\n\nprint(models[-1](tf.keras.Input(shape=(543, 3))))","metadata":{"execution":{"iopub.status.busy":"2023-03-14T22:46:16.328699Z","iopub.execute_input":"2023-03-14T22:46:16.329788Z","iopub.status.idle":"2023-03-14T22:46:21.858336Z","shell.execute_reply.started":"2023-03-14T22:46:16.329750Z","shell.execute_reply":"2023-03-14T22:46:21.857351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf_model_path = \"/kaggle/input/asl-sign-detection-pytorch-lightning/tf_model\"","metadata":{"execution":{"iopub.status.busy":"2023-03-14T22:46:21.860976Z","iopub.execute_input":"2023-03-14T22:46:21.861824Z","iopub.status.idle":"2023-03-14T22:46:21.866429Z","shell.execute_reply.started":"2023-03-14T22:46:21.861784Z","shell.execute_reply":"2023-03-14T22:46:21.865391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DROP_Z = False\n\nNUM_FRAMES = 15\nSEGMENTS = 3\n\nLEFT_HAND_OFFSET = 468\nPOSE_OFFSET = LEFT_HAND_OFFSET + 21\nRIGHT_HAND_OFFSET = POSE_OFFSET + 33\n\n## average over the entire face, and the entire 'pose'\naveraging_sets = [[0, 468], [POSE_OFFSET, 33]]\n\nlip_landmarks = [\n 61,\n 185,\n 40,\n 39,\n 37,\n 0,\n 267,\n 269,\n 270,\n 409,\n 291,\n 146,\n 91,\n 181,\n 84,\n 17,\n 314,\n 405,\n 321,\n 375,\n 78,\n 191,\n 80,\n 81,\n 82,\n 13,\n 312,\n 311,\n 310,\n 415,\n 95,\n 88,\n 178,\n 87,\n 14,\n 317,\n 402,\n 318,\n 324,\n 308,\n]\nleft_hand_landmarks = list(range(LEFT_HAND_OFFSET, LEFT_HAND_OFFSET + 21))\nright_hand_landmarks = list(range(RIGHT_HAND_OFFSET, RIGHT_HAND_OFFSET + 21))\n\npoint_landmarks = [\n item\n for sublist in [lip_landmarks, left_hand_landmarks, right_hand_landmarks]\n for item in sublist\n]\n\nLANDMARKS = len(point_landmarks) + len(averaging_sets)\nprint(LANDMARKS)\nif DROP_Z:\n INPUT_SHAPE = (NUM_FRAMES, LANDMARKS * 2)\nelse:\n INPUT_SHAPE = (NUM_FRAMES, LANDMARKS * 3)\n\nFLAT_INPUT_SHAPE = (INPUT_SHAPE[0] + 2 * (SEGMENTS + 1)) * INPUT_SHAPE[1]","metadata":{"execution":{"iopub.status.busy":"2023-03-14T22:46:21.867792Z","iopub.execute_input":"2023-03-14T22:46:21.868509Z","iopub.status.idle":"2023-03-14T22:46:21.882652Z","shell.execute_reply.started":"2023-03-14T22:46:21.868470Z","shell.execute_reply":"2023-03-14T22:46:21.881475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def tf_nan_mean(x, axis=0):\n return tf.reduce_sum(\n tf.where(tf.math.is_nan(x), tf.zeros_like(x), x), axis=axis\n ) / tf.reduce_sum(\n tf.where(tf.math.is_nan(x), tf.zeros_like(x), tf.ones_like(x)), axis=axis\n )\n\n\ndef tf_nan_std(x, axis=0):\n d = x - tf_nan_mean(x, axis=axis)\n return tf.math.sqrt(tf_nan_mean(d * d, axis=axis))\n\n\ndef flatten_means_and_stds(x, axis=0):\n # Get means and stds\n x_mean = tf_nan_mean(x, axis=0)\n x_std = tf_nan_std(x, axis=0)\n\n x_out = tf.concat([x_mean, x_std], axis=0)\n x_out = tf.reshape(x_out, (1, INPUT_SHAPE[1] * 2))\n x_out = tf.where(tf.math.is_finite(x_out), x_out, tf.zeros_like(x_out))\n return x_out","metadata":{"execution":{"iopub.status.busy":"2023-03-14T22:46:21.884482Z","iopub.execute_input":"2023-03-14T22:46:21.884835Z","iopub.status.idle":"2023-03-14T22:46:21.896095Z","shell.execute_reply.started":"2023-03-14T22:46:21.884795Z","shell.execute_reply":"2023-03-14T22:46:21.895196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class FeatureGenTF(tf.keras.layers.Layer):\n def __init__(self):\n super().__init__()\n\n def call(self, x_in):\n if DROP_Z:\n x_in = x_in[:, :, 0:2]\n x_list = [\n tf.expand_dims(\n tf_nan_mean(x_in[:, av_set[0] : av_set[0] + av_set[1], :], axis=1),\n axis=1,\n )\n for av_set in averaging_sets\n ]\n x_list.append(tf.gather(x_in, point_landmarks, axis=1))\n x = tf.concat(x_list, 1)\n\n x_padded = x\n for i in range(SEGMENTS):\n p0 = tf.where(\n ((tf.shape(x_padded)[0] % SEGMENTS) > 0) & ((i % 2) != 0), 1, 0\n )\n p1 = tf.where(\n ((tf.shape(x_padded)[0] % SEGMENTS) > 0) & ((i % 2) == 0), 1, 0\n )\n paddings = [[p0, p1], [0, 0], [0, 0]]\n x_padded = tf.pad(x_padded, paddings, mode=\"SYMMETRIC\")\n x_list = tf.split(x_padded, SEGMENTS)\n x_list = [flatten_means_and_stds(_x, axis=0) for _x in x_list]\n\n x_list.append(flatten_means_and_stds(x, axis=0))\n\n ## Resize only dimension 0. Resize can't handle nan, so replace nan with that dimension's avg value to reduce impact.\n x = tf.image.resize(\n tf.where(tf.math.is_finite(x), x, tf_nan_mean(x, axis=0)),\n [NUM_FRAMES, LANDMARKS],\n )\n x = tf.reshape(x, (1, INPUT_SHAPE[0] * INPUT_SHAPE[1]))\n x = tf.where(tf.math.is_nan(x), tf.zeros_like(x), x)\n x_list.append(x)\n x = tf.concat(x_list, axis=1)\n return x","metadata":{"execution":{"iopub.status.busy":"2023-03-14T22:46:21.897434Z","iopub.execute_input":"2023-03-14T22:46:21.897865Z","iopub.status.idle":"2023-03-14T22:46:21.911269Z","shell.execute_reply.started":"2023-03-14T22:46:21.897828Z","shell.execute_reply":"2023-03-14T22:46:21.910300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class ASLInferModel(tf.keras.layers.Layer):\n def __init__(self, tf_model_path):\n super().__init__()\n\n self.feature_gen = FeatureGenTF()\n self.model = tf.saved_model.load(tf_model_path)\n self.feature_gen.trainable = False\n self.model.trainable = False\n\n @tf.function(\n input_signature=[\n tf.TensorSpec(shape=[None, 543, 3], dtype=tf.float32, name=\"inputs\")\n ]\n )\n def call(self, inputs):\n features = self.feature_gen(tf.cast(inputs, dtype=tf.float32))\n outputs = self.model(input=features)\n return outputs\n\n\nmytfmodel = ASLInferModel(tf_model_path)","metadata":{"execution":{"iopub.status.busy":"2023-03-14T22:46:21.913237Z","iopub.execute_input":"2023-03-14T22:46:21.914205Z","iopub.status.idle":"2023-03-14T22:46:22.721946Z","shell.execute_reply.started":"2023-03-14T22:46:21.914152Z","shell.execute_reply":"2023-03-14T22:46:22.720758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class TFLiteEnsemble(tf.keras.Model):\n def __init__(self, models, weights):\n super(TFLiteEnsemble, self).__init__()\n self.weight_list = weights\n self.models = models\n \n @tf.function(input_signature=[tf.TensorSpec(shape=[None, 543, 3], dtype=tf.float32, name='inputs')])\n def call(self, inputs):\n x = tf.cast(inputs, dtype=tf.float32)\n ## The commented out line might work better for KFold, I haven't tested it yet.\n # model_outputs = [model(x) for model in self.models]\n\n ## This version with '.popitem()[1]' requires that the 'model(inputs)' call always returns a dict with only one key: value pair.\n ## It is a workaround due to different models having dict['output'] vs dict['outputs']\n ## Second workaround: tf.reshape to handle either (1, 250) or (250)\n model_outputs = [tf.reshape(model(x).popitem()[1], [-1]) for model in self.models]\n ## NOTE: this assumes weights add to 1.0!\n outputs = tf.add_n([tf.multiply(w, inp) for (w, inp) in zip(self.weight_list, model_outputs)])\n\n # Return a dictionary with the output tensor\n return {'outputs': outputs}\n\nensemble_model = TFLiteEnsemble(models, weights)","metadata":{"execution":{"iopub.status.busy":"2023-03-14T22:46:22.723617Z","iopub.execute_input":"2023-03-14T22:46:22.724010Z","iopub.status.idle":"2023-03-14T22:46:22.739667Z","shell.execute_reply.started":"2023-03-14T22:46:22.723968Z","shell.execute_reply":"2023-03-14T22:46:22.738506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"keras_model_converter = tf.lite.TFLiteConverter.from_keras_model(ensemble_model)\ntflite_model = keras_model_converter.convert()\nwith open('model.tflite', 'wb') as f:\n f.write(tflite_model)\n!zip submission.zip model.tflite\n\nprint(\"done\")","metadata":{"execution":{"iopub.status.busy":"2023-03-14T22:46:22.743879Z","iopub.execute_input":"2023-03-14T22:46:22.744527Z","iopub.status.idle":"2023-03-14T22:46:34.227308Z","shell.execute_reply.started":"2023-03-14T22:46:22.744497Z","shell.execute_reply":"2023-03-14T22:46:34.225871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ROWS_PER_FRAME = 543 # number of landmarks per frame\ndef load_relevant_data_subset(pq_path):\n data_columns = ['x', 'y', 'z']\n data = pd.read_parquet(pq_path, columns=data_columns)\n n_frames = int(len(data) / ROWS_PER_FRAME)\n data = data.values.reshape(n_frames, ROWS_PER_FRAME, len(data_columns))\n return data.astype(np.float32)","metadata":{"execution":{"iopub.status.busy":"2023-03-14T22:46:34.229631Z","iopub.execute_input":"2023-03-14T22:46:34.230047Z","iopub.status.idle":"2023-03-14T22:46:34.237736Z","shell.execute_reply.started":"2023-03-14T22:46:34.230001Z","shell.execute_reply":"2023-03-14T22:46:34.236394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Manually add decoder\ndecoder = {v:k for k, v in {'tv': 0, 'after': 1, 'airplane': 2, 'all': 3, 'alligator': 4, 'animal': 5, 'another': 6, 'any': 7, 'apple': 8, 'arm': 9, 'aunt': 10, 'awake': 11, 'backyard': 12, 'bad': 13, 'balloon': 14, \n 'bath': 15, 'because': 16, 'bed': 17, 'bedroom': 18, 'bee': 19, 'before': 20, 'beside': 21, 'better': 22, 'bird': 23, 'black': 24, 'blow': 25, 'blue': 26, 'boat': 27, 'book': 28, 'boy': 29, \n 'brother': 30, 'brown': 31, 'bug': 32, 'bye': 33, 'callonphone': 34, 'can': 35, 'car': 36, 'carrot': 37, 'cat': 38, 'cereal': 39, 'chair': 40, 'cheek': 41, 'child': 42, 'chin': 43, 'chocolate': 44, \n 'clean': 45, 'close': 46, 'closet': 47, 'cloud': 48, 'clown': 49, 'cow': 50, 'cowboy': 51, 'cry': 52, 'cut': 53, 'cute': 54, 'dad': 55, 'dance': 56, 'dirty': 57, 'dog': 58, 'doll': 59, 'donkey': 60, \n 'down': 61, 'drawer': 62, 'drink': 63, 'drop': 64, 'dry': 65, 'dryer': 66, 'duck': 67, 'ear': 68, 'elephant': 69, 'empty': 70, 'every': 71, 'eye': 72, 'face': 73, 'fall': 74, 'farm': 75, 'fast': 76, \n 'feet': 77, 'find': 78, 'fine': 79, 'finger': 80, 'finish': 81, 'fireman': 82, 'first': 83, 'fish': 84, 'flag': 85, 'flower': 86, 'food': 87, 'for': 88, 'frenchfries': 89, 'frog': 90, 'garbage': 91, \n 'gift': 92, 'giraffe': 93, 'girl': 94, 'give': 95, 'glasswindow': 96, 'go': 97, 'goose': 98, 'grandma': 99, 'grandpa': 100, 'grass': 101, 'green': 102, 'gum': 103, 'hair': 104, 'happy': 105, 'hat': 106, \n 'hate': 107, 'have': 108, 'haveto': 109, 'head': 110, 'hear': 111, 'helicopter': 112, 'hello': 113, 'hen': 114, 'hesheit': 115, 'hide': 116, 'high': 117, 'home': 118, 'horse': 119, 'hot': 120, 'hungry': 121, \n 'icecream': 122, 'if': 123, 'into': 124, 'jacket': 125, 'jeans': 126, 'jump': 127, 'kiss': 128, 'kitty': 129, 'lamp': 130, 'later': 131, 'like': 132, 'lion': 133, 'lips': 134, 'listen': 135, 'look': 136, \n 'loud': 137, 'mad': 138, 'make': 139, 'man': 140, 'many': 141, 'milk': 142, 'minemy': 143, 'mitten': 144, 'mom': 145, 'moon': 146, 'morning': 147, 'mouse': 148, 'mouth': 149, 'nap': 150, 'napkin': 151, \n 'night': 152, 'no': 153, 'noisy': 154, 'nose': 155, 'not': 156, 'now': 157, 'nuts': 158, 'old': 159, 'on': 160, 'open': 161, 'orange': 162, 'outside': 163, 'owie': 164, 'owl': 165, 'pajamas': 166, 'pen': 167,\n 'pencil': 168, 'penny': 169, 'person': 170, 'pig': 171, 'pizza': 172, 'please': 173, 'police': 174, 'pool': 175, 'potty': 176, 'pretend': 177, 'pretty': 178, 'puppy': 179, 'puzzle': 180, 'quiet': 181, 'radio': 182, \n 'rain': 183, 'read': 184, 'red': 185, 'refrigerator': 186, 'ride': 187, 'room': 188, 'sad': 189, 'same': 190, 'say': 191, 'scissors': 192, 'see': 193, 'shhh': 194, 'shirt': 195, 'shoe': 196, 'shower': 197, \n 'sick': 198, 'sleep': 199, 'sleepy': 200, 'smile': 201, 'snack': 202, 'snow': 203, 'stairs': 204, 'stay': 205, 'sticky': 206, 'store': 207, 'story': 208, 'stuck': 209, 'sun': 210, 'table': 211, 'talk': 212, \n 'taste': 213, 'thankyou': 214, 'that': 215, 'there': 216, 'think': 217, 'thirsty': 218, 'tiger': 219, 'time': 220, 'tomorrow': 221, 'tongue': 222, 'tooth': 223, 'toothbrush': 224, 'touch': 225, 'toy': 226, \n 'tree': 227, 'uncle': 228, 'underwear': 229, 'up': 230, 'vacuum': 231, 'wait': 232, 'wake': 233, 'water': 234, 'wet': 235, 'weus': 236, 'where': 237, 'white': 238, 'who': 239, 'why': 240, 'will': 241, \n 'wolf': 242, 'yellow': 243, 'yes': 244, 'yesterday': 245, 'yourself': 246, 'yucky': 247, 'zebra': 248, 'zipper': 249}.items()}","metadata":{"execution":{"iopub.status.busy":"2023-03-14T22:46:34.239356Z","iopub.execute_input":"2023-03-14T22:46:34.239649Z","iopub.status.idle":"2023-03-14T22:46:34.262498Z","shell.execute_reply.started":"2023-03-14T22:46:34.239622Z","shell.execute_reply":"2023-03-14T22:46:34.261300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install tflite-runtime\nimport tflite_runtime.interpreter as tflite\n\ninterpreter = tflite.Interpreter(\"model.tflite\")\nfound_signatures = list(interpreter.get_signature_list().keys())\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\n\noutput = prediction_fn(inputs=load_relevant_data_subset(f'/kaggle/input/asl-signs/{pd.read_csv(\"/kaggle/input/asl-signs/train.csv\").path[0]}'))\nsign = np.argmax(output[\"outputs\"])\n\nprint(\"PRED : \", decoder[sign])\nprint(\"GT : \", pd.read_csv(\"/kaggle/input/asl-signs/train.csv\").sign[0])","metadata":{"execution":{"iopub.status.busy":"2023-03-14T22:46:34.264420Z","iopub.execute_input":"2023-03-14T22:46:34.264781Z","iopub.status.idle":"2023-03-14T22:46:45.820945Z","shell.execute_reply.started":"2023-03-14T22:46:34.264743Z","shell.execute_reply":"2023-03-14T22:46:45.818887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Every day, 33 babies are born with permanent hearing loss in the U.S.**\n\nAround 90% of which are born to hearing parents many of which may not know American Sign Language. (kdhe.ks.gov, deafchildren.org) Without sign language, deaf babies are at risk of Language Deprivation Syndrome. This syndrome is characterized by a lack of access to naturally occurring language acquisition during their critical language-learning years. It can cause serious impacts on different aspects of their lives, such as relationships, education, and employment.\n\n**Learning sign language is challenging.**\n\nLearning American Sign Language is as difficult for English speakers as learning Japanese. (jstor.org) It takes time and resources, which many parents don't have. They want to learn sign language, but it's hard when they are working long hours just to make ends meet. And even if they find the time and money for classes, the classes are often far away.\n\n**Games can help.**\n\nPopSign is a smartphone game app that makes learning American Sign Language fun, interactive, and accessible. Players match videos of ASL signs with bubbles containing written English words to pop them.\nPopSign is designed to help parents with deaf children learn ASL, but it's open to anyone who wants to learn sign language vocabulary. By adding a sign language recognizer from this competition, PopSign players will be able to sign the type of bubble they want to shoot, providing the player with the opportunity to practice the sign themselves instead of just watching videos of other people signing.\n\n**You can help connect deaf children and their parents.**\n\nBy training a sign language recognizer for PopSign, you can help make the game more interactive and improve the learning and confidence of players who want to learn sign language to communicate with their loved ones.","metadata":{}}]}