{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n\n
\n\n
\n

\n
\"Google

Isolated Sign Language Recognition

Exploratory     Data     Analysis

\n \n
\"asl
\n
\n\n
\n","metadata":{}},{"cell_type":"markdown","source":"\n\n![image.png](attachment:7e4d3247-cc0d-4ff4-9772-0fde74687ce8.png)\n\n1. [Importng Libraries and Setting File Directories](#section-one)\n2. [Data Visualization](#section-two)\n3. [Loading Data](#section-three)\n4. [Model Training](#section-four)\n5. [Model Submission](#section-five)","metadata":{"execution":{"iopub.status.busy":"2023-03-19T10:51:52.612172Z","iopub.execute_input":"2023-03-19T10:51:52.613569Z","iopub.status.idle":"2023-03-19T10:51:52.624031Z","shell.execute_reply.started":"2023-03-19T10:51:52.613506Z","shell.execute_reply":"2023-03-19T10:51:52.622518Z"}}},{"cell_type":"markdown","source":"
\n\n

1) INTRODUCTION

\n\n---\n\n* This notebook is created for [Kaggle's Isolated Sign Language Recognition Competition](https://www.kaggle.com/competitions/asl-signs/overview)\n\n* It is a compilation of various resources which have been modified to create this notebook. Please acknowledge and give upvotes to these resources if you find them useful.\n\n* The notebook [🤟 GISLR 🤟 - 📚Learn – 🔭EDA – 🤖Baseline](https://www.kaggle.com/code/dschettler8845/gislr-learn-eda-baseline#introduction) was used as the major reference..\n\n* This notebook is designed with beginners in mind, and includes detailed descriptions to aid in learning the model.","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

2)  BACKGROUND INFORMATION    

","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.1 OVERVIEW

\n\n---\n\n
PRIMARY TASK DESCRIPTION\n\n

The Isolated Sign Language Recognition competition's goal is to classify isolated American Sign Language (ASL) signs. You will create a TensorFlow Lite model trained on labeled landmark data extracted using the MediaPipe Holistic Solution.

\n\nThe evaluation metric for this contest is simple classification accuracy\n\n\n
IMPORTANT RELEVANT TERMS\n\n\n\n
\n

Why TensorFlow Lite

\n

To allow the ML model to run on device in an attempt to limit latency inside the game, PopSign doesn’t send user videos to the cloud. Therefore, all inference must be done on the phone itself. PopSign is building its recognition pipeline on top of TensorFlow Lite, which runs on both Android and iOS. In order for the competition models to integrate seamlessly with PopSign, we are asking our competitors to submit their entries in the form of TensorFlow Lite models.

\n
\n\n
COMPETITION IMPACT INFORMATION\n\n\n\n
COMPETITION HOST AND CONTRIBUTER INFORMATION\n\n\n\n
\n\n
VISUAL EXPLANATION\n\n

\"video\"

\n\n
","metadata":{}},{"cell_type":"markdown","source":"
American Sign Language (ASL)\n\n
\n\nTextbook Definition (Key Points) [REF]\n\n\n
\n\nELI5 Competition Definition\n\nAmerican Sign Language (ASL) is a language used primarily by members of the Deaf community in North America. It is not a visual representation of English, but has its own unique grammar, syntax, and vocabulary. ASL is a visual-gestural language, meaning that it uses facial expressions, body language, and hand movements to convey meaning.\n\n
Explain With Pictures\n\nCute Animated ASL GIF ...\n\n\n\n\n
\n
MediaPipe Holistic Solution\n\n
\n\nTextbook Definition (Key Points) [REF]\n\n\n\n
\n\nELI5 Competition Definition\n\nMediaPipe Holistic Solution is a powerful, easy-to-use software tool that can detect and track multiple human body parts and gestures in real-time video streams. It is open-source and can run on a variety of platforms, including mobile devices, making it an ideal solution for our competition.\n\nWe will have access to landmark data provided by Mediapipe for hands and face.\n\n
Explain With Pictures\n\nExample of MediaPipe Holistic [REF]\n\n\n\n
\n\nMediaPipe Landmarks for Hands [REF]\n\n\n\n
\n\n---\n\n
\n\n
TensorFlow Lite\n\n
\n\nTextbook Definition (Key Points) [REF]\n\n\n\n\n\n
\n\nELI5 Competition Definition\n\nTensorFlow Lite is a specialized version of TensorFlow, a powerful machine learning framework that has been optimized for running on mobile and embedded devices. It allows us to run our models on a variety of devices, including smartphones and tablets, and provides fast, low-latency performance.\n\n
Explain With Pictures\n\nTF Model Conversion Workflow\n\n\n\n
\n\n---\n\n
\n\n
Isolated Sign Language Recognition (ISLR) [REF]\n\n
\n\nTextbook Definition (Key Points)\n\n\n\n
\n\nELI5 Competition Definition\n\nISLR stands for Isolated Sign Language Recognition. It is a technology that uses machine learning to help computers understand and interpret sign language.\n\nBasically, ISLR teaches computers to recognize and differentiate between different hand gestures used in sign language, so that it can then interpret what the person is saying. This is a big deal because it can help people who are deaf or hard of hearing communicate more easily with computers and technology.\n\nIn this competition, the task is to train a machine learning model that can accurately recognize isolated sign language signs and classify them into the correct sign category.\n\n
Explain With Pictures\n\nAmerican Sign Language Hand Gestures in Isolation\n\n\n\n
\n\n---\n\n
\n\n
Continuous Sign Language Recognition (CSLR)\n\n
\n\nTextbook Definition (Key Points) [REF]\n\n\n\n
\n\nELI5 Competition Definition\n\nWhile we are not performing CSLR (Continuous Sign Language Recognition) in this competition it is important to compare and contrast it with ISLR (Isolated Sign Language Recognition).\n\nCSLR (Continuous Sign Language Recognition) is like ISLR (Isolated Sign Language Recognition), but instead of recognizing individual signs in isolation, it recognizes signs as part of a continuous sentence or phrase. This requires the recognition system to be able to handle more complex, continuous motion and to understand the context and grammar of sign language.\n\nImagine trying to understand what someone is saying to you when they don't stop talking, and you don't know where one word ends and the next one begins. CSLR tries to do the same thing, but with sign language instead of spoken language. It's like trying to listen to a long sentence and figure out the meaning, but with your eyes instead of your ears.\n\n
Explain With Pictures\n\nTBD\n\n\n\n
\n\n---\n\n
\n\n
Landmark data\n\n
\n\nTextbook Definition (Key Points) [REF is the competition for now]\n\n\n\n
\n\nELI5 Competition Definition\n\nLandmarks or keypoints are like dots that are placed on important areas of an object or a person's body. These dots help a computer to understand where these important areas are and how they are moving.\n\nIn the context of ISLR and MediaPipe, landmarks/keypoints are used to help a computer understand the movements of a person's hands and body when they are signing in sign language. By tracking the movements of these landmarks/keypoints, the computer can then recognize which sign the person is making.\n\nUsing keypoints/landmarks is way less computationally expensive than using video or images.\n\n
Explain With Pictures\n\nIn-browser touchless control demos. Left: Palm picker, touch interface, keyboard. Right: Distant touchless keyboard. Try it out!\n\n\n\n
\n\n---\n\n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

3)  IMPORTS    

","metadata":{}},{"cell_type":"markdown","source":"The first block of codes checks if `tensorflow-io` and `mediapipe` are installed, if not, then it installs them using `pip`. \n\nThe second block of codes imports various Python libraries that will be used in the rest of the program. These imports include machine learning and data science libraries such as `tensorflow`, `tensorflow_io`, `pandas`, `numpy`, and `sklearn`. Other imports include visualization libraries such as `matplotlib`, `plotly`, `seaborn`, and `cv2`. \n\nThe third block of codes defines a function called `seed_it_all()` which sets specific environmental variables and seeds to make the code more reproducible. \n\nOverall, these codes handle the installation of necessary dependencies and imports required libraries for the program.","metadata":{}},{"cell_type":"code","source":"print(\"\\n... PIP INSTALLS STARTING ...\\n\")\n!pip install -q --upgrade tensorflow-io\ntry:\n import mediapipe as mp\nexcept:\n !pip install -q mediapipe\n import mediapipe as mp\nprint(\"\\n... PIP INSTALLS COMPLETE ...\\n\")\n\n\nprint(\"\\n... IMPORTS STARTING ...\\n\")\nprint(\"\\n\\tVERSION INFORMATION\")\n# Competition Specific Imports (You'll see why we need these later)\n# mediapipe above\n\n# Machine Learning and Data Science Imports (basics)\nimport tensorflow as tf; print(f\"\\t\\t– TENSORFLOW VERSION: {tf.__version__}\");\nimport tensorflow_io as tfio; print(f\"\\t\\t– TENSORFLOW-IO VERSION: {tfio.__version__}\");\nimport pandas as pd; pd.options.mode.chained_assignment = None; pd.set_option('display.max_columns', None);\nimport numpy as np; print(f\"\\t\\t– NUMPY VERSION: {np.__version__}\");\nimport sklearn; print(f\"\\t\\t– SKLEARN VERSION: {sklearn.__version__}\");\n\n# Built-In Imports (mostly don't worry about these)\nfrom kaggle_datasets import KaggleDatasets\nfrom collections import Counter\nfrom datetime import datetime\nfrom zipfile import ZipFile\nfrom glob import glob\nimport Levenshtein\nimport warnings\nimport requests\nimport hashlib\nimport imageio\nimport IPython\nimport sklearn\nimport urllib\nimport zipfile\nimport pickle\nimport random\nimport shutil\nimport string\nimport json\nimport math\nimport time\nimport gzip\nimport ast\nimport sys\nimport io\nimport os\nimport gc\nimport re\n\n# Visualization Imports (overkill)\nfrom matplotlib.animation import FuncAnimation\nfrom matplotlib.colors import ListedColormap\nfrom matplotlib.patches import Rectangle\nimport matplotlib.patches as patches\nimport plotly.graph_objects as go\nfrom IPython.display import HTML\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm; tqdm.pandas();\nimport plotly.express as px\nimport tifffile as tif\nimport seaborn as sns\nfrom PIL import Image, ImageEnhance; Image.MAX_IMAGE_PIXELS = 5_000_000_000;\nimport matplotlib; print(f\"\\t\\t– MATPLOTLIB VERSION: {matplotlib.__version__}\");\nfrom matplotlib import animation, rc; rc('animation', html='jshtml')\nimport plotly\nimport PIL\nimport cv2\n\nimport plotly.io as pio\nprint(pio.renderers)\n\ndef seed_it_all(seed=7):\n \"\"\" Attempt to be Reproducible \"\"\"\n os.environ['PYTHONHASHSEED'] = str(seed)\n random.seed(seed)\n np.random.seed(seed)\n tf.random.set_seed(seed)\n\nseed_it_all()\n\nprint(\"\\n\\n... IMPORTS COMPLETE ...\\n\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-03-19T12:09:13.197477Z","iopub.execute_input":"2023-03-19T12:09:13.198413Z","iopub.status.idle":"2023-03-19T12:09:51.102644Z","shell.execute_reply.started":"2023-03-19T12:09:13.198344Z","shell.execute_reply":"2023-03-19T12:09:51.101311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

4)  SETUP AND HELPER FUNCTIONS    

","metadata":{"execution":{"iopub.status.busy":"2023-03-19T11:23:30.235881Z","iopub.execute_input":"2023-03-19T11:23:30.23627Z","iopub.status.idle":"2023-03-19T11:23:30.244312Z","shell.execute_reply.started":"2023-03-19T11:23:30.236235Z","shell.execute_reply":"2023-03-19T11:23:30.242701Z"}}},{"cell_type":"markdown","source":"
\n\n

4.0 FUNCTIONS FROM OTHER KAGGLERS!

\n\n---\n\n
\n\nAnimation Function(s)","metadata":{}},{"cell_type":"markdown","source":"# Visualizing things is always enjoyable, isn't it? \nHowever, since the data we are dealing with pertains to sequential hand movements, it may not be easily perceptible through tabular presentation alone. Therefore, it would be preferable to represent the movements using matplotlib through plotting. But before that, let's take a look at all different landmarks that are available in the dataset according to mediapipe's definitions to better understand the data in any particular frame.\n\n**Landmark Models\nMediaPipe Holistic utilizes the pose, face and hand landmark models in MediaPipe Pose, MediaPipe Face Mesh and MediaPipe Hands respectively to generate a total of 543 landmarks (33 pose landmarks, 468 face landmarks, and 21 hand landmarks per hand).**\n\n- **Face Landmarks = 468**\n\n![image.png](attachment:3a91005b-01f2-40ed-a230-35298fcef4ac.png)\n\n- **Pose Landmarks = 33**\n\n![image.png](attachment:6cd0f0ed-a58c-4721-836b-858ecf934e50.png)\n\n- **Hand Landmarks = 21**\n\n![image.png](attachment:4f1cf651-9811-4f1a-aa27-4982ec183595.png)\n\nTotal landmarks `= 468 + 33 + 21(left) + 21(right) = 543`","metadata":{"execution":{"iopub.status.busy":"2023-03-19T11:23:45.519781Z","iopub.execute_input":"2023-03-19T11:23:45.520211Z","iopub.status.idle":"2023-03-19T11:23:45.539555Z","shell.execute_reply.started":"2023-03-19T11:23:45.520175Z","shell.execute_reply":"2023-03-19T11:23:45.537897Z"}}},{"cell_type":"code","source":"def get_hand_points(hand):\n \"\"\"Return x, y lists of normalized spatial coordinates for each finger in the hand dataframe.\"\"\"\n def __get_hand_ax(_axis):\n return [np.nan_to_num(_x) for _x in \n [hand.iloc[i][_axis] for i in range(5)]+\\\n [[hand.iloc[i][_axis] for i in range(j, j+4)] for j in range(5, 21, 4)]+\\\n [hand.iloc[i][_axis] for i in special_pts]]\n special_pts = [0, 5, 9, 13, 17, 0]\n return [__get_hand_ax(_ax) for _ax in ['x','y','z']]\n\ndef get_pose_points(pose):\n \"\"\"\n Extracts x and y coordinates from the provided dataframe for pose landmarks.\n\n Args:\n pose (pandas.DataFrame): Dataframe containing pose landmarks with columns ['x', 'y', 'z', 'visibility', 'presence'].\n\n Returns:\n tuple: Two lists of x and y coordinates, respectively.\n\n \"\"\"\n def __get_pose_ax(_axis):\n return [np.nan_to_num(_x) for _x in [\n [pose.iloc[i][_axis] for i in [8, 6, 5, 4, 0, 1, 2, 3, 7]], \n [pose.iloc[i][_axis] for i in [10, 9]], \n [pose.iloc[i][_axis] for i in [22, 16, 20, 18, 16, 14, 12, 11, 13, 15, 17, 19, 15, 21]], \n [pose.iloc[i][_axis] for i in [12, 24, 26, 28, 30, 32, 28]], \n [pose.iloc[i][_axis] for i in [11, 23, 25, 27, 29, 31, 27]], \n [pose.iloc[i][_axis] for i in [24, 23]]\n ]]\n return [__get_pose_ax(_ax) for _ax in ['x','y','z']]\n\n\ndef animation_frame(f, event_df, ax, ax_pad=0.2, style=\"full\", \n face_color=\"spring\", pose_color=\"autumn\", lh_color=\"winter\", rh_color=\"summer\"):\n \"\"\"\n Function called by FuncAnimation to animate the plot with the provided frame.\n\n Args:\n f (int): The current frame number.\n\n Returns:\n None.\n \"\"\"\n \n face_color = plt.cm.get_cmap(face_color)\n pose_color = plt.cm.get_cmap(pose_color)\n rh_color = plt.cm.get_cmap(rh_color)\n lh_color = plt.cm.get_cmap(lh_color)\n \n sign_df = event_df.copy()\n \n # Clear axis and fix the axis\n ax.clear()\n if style==\"full\":\n xmin = sign_df['x'].min() - ax_pad\n xmax = sign_df['x'].max() + ax_pad\n ymin = sign_df['y'].min() - ax_pad\n ymax = sign_df['y'].max() + ax_pad\n elif style==\"hands\":\n xmin = sign_df[sign_df.type.isin([\"left_hand\", \"right_hand\"])]['x'].min() - ax_pad\n xmax = sign_df[sign_df.type.isin([\"left_hand\", \"right_hand\"])]['x'].max() + ax_pad\n ymin = sign_df[sign_df.type.isin([\"left_hand\", \"right_hand\"])]['y'].min() - ax_pad\n ymax = sign_df[sign_df.type.isin([\"left_hand\", \"right_hand\"])]['y'].max() + ax_pad\n else:\n xmin = sign_df[sign_df.type==style]['x'].min() - ax_pad\n xmax = sign_df[sign_df.type==style]['x'].max() + ax_pad\n ymin = sign_df[sign_df.type==style]['y'].min() - ax_pad\n ymax = sign_df[sign_df.type==style]['y'].max() + ax_pad\n \n ax.set_xlim(xmin, xmax)\n ax.set_ylim(ymin, ymax)\n ax.axis(False) # Remove the axis lines\n \n # Normalize depth\n zmin, zmax = sign_df['z'].min(), sign_df['z'].max()\n sign_df['z'] = (sign_df['z']-zmin)/(zmax-zmin)\n \n # Get data for current frame\n frame = sign_df[sign_df.frame==f]\n \n # Left Hand\n if style.lower() in [\"left_hand\", \"hands\", \"full\"]:\n left = frame[frame.type=='left_hand']\n lx, ly, lz = get_hand_points(left)\n for i in range(len(lx)):\n if type(lx[i])!=np.float64:\n lh_clr = [lh_color(((np.abs(_x)+np.abs(_y))/2)) for _x, _y in zip(lx[i], ly[i])]\n lh_clr = tuple(sum(_x)/len(_x) for _x in zip(*lh_clr))\n else: \n lh_clr = lh_color(((np.abs(lx[i])+np.abs(ly[i]))/2))\n ax.plot(lx[i], ly[i], color=lh_clr, alpha=lz[i].mean())\n \n # Right Hand\n if style.lower() in [\"right_hand\", \"hands\", \"full\"]:\n right = frame[frame.type=='right_hand']\n rx, ry, rz = get_hand_points(right)\n for i in range(len(rx)):\n if type(rx[i])!=np.float64:\n rh_clr = [rh_color((np.abs(_x)+np.abs(_y))/2) for _x, _y in zip(rx[i], ry[i])] \n rh_clr = tuple(sum(_x)/len(_x) for _x in zip(*rh_clr))\n else:\n rh_clr = rh_color(((np.abs(rx[i])+np.abs(ry[i]))/2))\n ax.plot(rx[i], ry[i], color=rh_clr, alpha=rz[i].mean())\n \n # Pose\n if style.lower() in [\"pose\", \"full\"]:\n pose = frame[frame.type=='pose']\n px, py, pz = get_pose_points(pose)\n for i in range(len(px)):\n if type(px[i])!=np.float64:\n pose_clr = [pose_color(((np.abs(_x)+np.abs(_y))/2)) for _x, _y in zip(px[i], py[i])]\n pose_clr = tuple(sum(_x)/len(_x) for _x in zip(*pose_clr))\n else: \n pose_clr = pose_color(((np.abs(px[i])+np.abs(py[i]))/2))\n ax.plot(px[i], py[i], color=pose_clr, alpha=pz[i].mean())\n \n if style.lower() in [\"face\", \"full\"]:\n face = frame[frame.type=='face'][['x', 'y', 'z']].values\n fx, fy, fz = face[:,0], face[:,1], face[:,2]\n for i in range(len(fx)):\n ax.plot(fx[i], fy[i], '.', color=pose_color(fz[i]), alpha=fz[i])\n \n # Use this so we don't get an extra return\n plt.close()\n \n \ndef plot_event(event_df, style=\"full\"):\n # Create figure and animation\n fig, ax = plt.subplots()\n l, = ax.plot([], [])\n animation = FuncAnimation(fig, func=lambda x: animation_frame(x, event_df, ax, style=style), \n frames=event_df[\"frame\"].unique())\n \n # Display animation as HTML5 video\n return HTML(animation.to_html5_video())","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:09:51.105178Z","iopub.execute_input":"2023-03-19T12:09:51.105654Z","iopub.status.idle":"2023-03-19T12:09:51.14194Z","shell.execute_reply.started":"2023-03-19T12:09:51.105606Z","shell.execute_reply":"2023-03-19T12:09:51.140634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below code defines functions to animate and plot different types of gesture events. \n\n- `get_hand_points(hand)` takes a pandas dataframe of hand landmarks and returns normalized spatial coordinates for each finger (x, y lists). \n- `get_pose_points(pose)` takes a pandas dataframe of pose landmarks and extracts x and y coordinates for different parts of the body (returned as two lists). \n- `animation_frame(f, event_df, ax, ax_pad=0.2, style=\"full\", face_color=\"spring\", pose_color=\"autumn\", lh_color=\"winter\", rh_color=\"summer\")` is the core animation function that is called by the `plot_event(event_df, style=\"full\")` function. It takes several arguments, including the current frame number, the event dataframe, the axis to plot on, the padding around the axis limits, the color maps to use for different parts of the body, and the type of visual style to use (\"full\", \"hands\", \"pose\", or \"face\"). It uses the `get_hand_points` and `get_pose_points` functions to extract spatial coordinates for the hands and body parts, respectively, and plots them using matplotlib. It returns an HTML5 video of the animation. \n\nThe `plot_event(event_df, style=\"full\")` function takes an event dataframe and a style argument, and returns an HTML5 video of the animation generated by `animation_frame`. It creates a new figure and animation object, and passes them to the `animation_frame` function via the `FuncAnimation` class. The `style` argument determines what type of visual style to use, with \"full\" showing all parts of the body, \"hands\" showing only the hands, \"pose\" showing only the body pose, and \"face\" showing only the facial landmarks.","metadata":{}},{"cell_type":"markdown","source":"# Helper Functions\n\nBelow code defines four functions and a constant:\n\n1. `flatten_l_o_l(nested_list)` - this function takes a nested list as input and returns a flattened list containing all the items from the input list of lists. It uses a list comprehension and two nested loops to iterate over each item in the nested list.\n2. `print_ln(symbol=\"-\", line_len=110, newline_before=False, newline_after=False)` - this function prints a horizontal line of a specified length and symbol. It takes four optional parameters: the symbol to use for the line (default is \"-\"), the length of the line in characters (default is 110), whether to print a newline character before the line (default is False), and whether to print a newline character after the line (default is False).\n3. `read_json_file(file_path)` - this function reads a JSON file and returns a dictionary object representing the JSON data. It uses the `json` module to load the JSON data from the file and handles two possible exceptions: `FileNotFoundError` if the specified file path does not exist, and `ValueError` if the file does not contain valid JSON data.\n4. `get_sign_df(pq_path, invert_y=True)` - this function reads a parquet file containing sign data and returns a Pandas DataFrame object. If the `invert_y` parameter is True (default), it multiplies the \"y\" column of the DataFrame by -1 to invert the y value.\n5. `load_relevant_data_subset(pq_path)` - this function reads a parquet file containing relevant data subset and returns a numpy array of the data. It loads only the 'x', 'y', and 'z' columns and reshapes the data into a 3D numpy array, with the first dimension representing the number of frames, the second dimension representing the number of landmarks per frame (543), and the third dimension representing the number of data columns (3). The resulting data array is cast to dtype `np.float32`.\n6. `ROWS_PER_FRAME` - this is a constant that specifies the number of landmarks per frame, which is used in the `load_relevant_data_subset` function to reshape the data.","metadata":{}},{"cell_type":"code","source":"def flatten_l_o_l(nested_list):\n \"\"\"Flatten a list of lists into a single list.\n\n Args:\n nested_list (list): \n – A list of lists (or iterables) to be flattened.\n\n Returns:\n list: A flattened list containing all items from the input list of lists.\n \"\"\"\n return [item for sublist in nested_list for item in sublist]\n\n\ndef print_ln(symbol=\"-\", line_len=110, newline_before=False, newline_after=False):\n \"\"\"Print a horizontal line of a specified length and symbol.\n\n Args:\n symbol (str, optional): \n – The symbol to use for the horizontal line\n line_len (int, optional): \n – The length of the horizontal line in characters\n newline_before (bool, optional): \n – Whether to print a newline character before the line\n newline_after (bool, optional): \n – Whether to print a newline character after the line\n \"\"\"\n if newline_before: print();\n print(symbol * line_len)\n if newline_after: print();\n \n \ndef read_json_file(file_path):\n \"\"\"Read a JSON file and parse it into a Python object.\n\n Args:\n file_path (str): The path to the JSON file to read.\n\n Returns:\n dict: A dictionary object representing the JSON data.\n \n Raises:\n FileNotFoundError: If the specified file path does not exist.\n ValueError: If the specified file path does not contain valid JSON data.\n \"\"\"\n try:\n # Open the file and load the JSON data into a Python object\n with open(file_path, 'r') as file:\n json_data = json.load(file)\n return json_data\n except FileNotFoundError:\n # Raise an error if the file path does not exist\n raise FileNotFoundError(f\"File not found: {file_path}\")\n except ValueError:\n # Raise an error if the file does not contain valid JSON data\n raise ValueError(f\"Invalid JSON data in file: {file_path}\")\n \ndef get_sign_df(pq_path, invert_y=True):\n sign_df = pd.read_parquet(pq_path)\n \n # y value is inverted (Thanks @danielpeshkov)\n if invert_y: sign_df[\"y\"] *= -1 \n \n return sign_df\n\n\nROWS_PER_FRAME = 543 # number of landmarks per frame\ndef load_relevant_data_subset(pq_path):\n data_columns = ['x', 'y', 'z']\n data = pd.read_parquet(pq_path, columns=data_columns)\n n_frames = int(len(data) / ROWS_PER_FRAME)\n data = data.values.reshape(n_frames, ROWS_PER_FRAME, len(data_columns))\n return data.astype(np.float32)","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:09:51.146039Z","iopub.execute_input":"2023-03-19T12:09:51.146308Z","iopub.status.idle":"2023-03-19T12:09:51.158992Z","shell.execute_reply.started":"2023-03-19T12:09:51.146283Z","shell.execute_reply":"2023-03-19T12:09:51.157887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.2 LOAD DATA

\n\n---\n","metadata":{"execution":{"iopub.status.busy":"2023-03-19T11:23:08.276526Z","iopub.execute_input":"2023-03-19T11:23:08.277018Z","iopub.status.idle":"2023-03-19T11:23:08.284435Z","shell.execute_reply.started":"2023-03-19T11:23:08.276976Z","shell.execute_reply":"2023-03-19T11:23:08.28304Z"}}},{"cell_type":"markdown","source":"This code sets up the basic data for the American Sign Language (ASL) recognition system. It defines the path to the root data directory and loads the train dataframe from a csv file. It also loads the sign to prediction index map from a json file, where the sign represents each ASL hand gesture and the prediction index represents its numerical code, and creates encoder and decoder functions to convert between sign and prediction index. \n\nAfter the data has been loaded, the code displays a demo sign/event dataframe for a specific row in the train dataframe (DEMO_ROW) and plots the event using the plot_event function, which is currently not functional. \n\nNote that there is an option to load an extended train dataframe if the LOAD_EXTENDED variable is set to True and the extended_train.csv file exists in the specified directory. Otherwise, the original train.csv file is used.","metadata":{}},{"cell_type":"code","source":"# Define the path to the root data directory\nDATA_DIR = \"/kaggle/input/asl-signs\"\nEXTEND_TRAIN_DIR = \"/kaggle/input/gislr-extended-train-dataframe\" \n\nprint(\"\\n... BASIC DATA SETUP STARTING ...\\n\")\nprint(\"\\n\\n... LOAD TRAIN DATAFRAME FROM CSV FILE ...\\n\")\n\nLOAD_EXTENDED = True\nif LOAD_EXTENDED and os.path.isfile(os.path.join(EXTEND_TRAIN_DIR, \"extended_train.csv\")):\n train_df = pd.read_csv(os.path.join(EXTEND_TRAIN_DIR, \"extended_train.csv\"))\nelse:\n train_df = pd.read_csv(os.path.join(DATA_DIR, \"train.csv\"))\n train_df[\"path\"] = DATA_DIR+\"/\"+train_df[\"path\"]\ndisplay(train_df)\n\nprint(\"\\n\\n... LOAD SIGN TO PREDICTION INDEX MAP FROM JSON FILE ...\\n\")\ns2p_map = {k.lower():v for k,v in read_json_file(os.path.join(DATA_DIR, \"sign_to_prediction_index_map.json\")).items()}\np2s_map = {v:k for k,v in read_json_file(os.path.join(DATA_DIR, \"sign_to_prediction_index_map.json\")).items()}\nencoder = lambda x: s2p_map.get(x.lower())\ndecoder = lambda x: p2s_map.get(x)\nprint(s2p_map)\n\nDEMO_ROW = 283\nprint(f\"\\n\\n... DEMO SIGN/EVENT DATAFRAME FOR ROW {DEMO_ROW} - SIGN={train_df.iloc[DEMO_ROW]['sign']} ...\\n\")\ndemo_sign_df = get_sign_df(train_df.iloc[DEMO_ROW][\"path\"])\ndisplay(demo_sign_df)\n\n# I messed this function up... will fix later\nplot_event(demo_sign_df)# Define the path to the root data directory\nDATA_DIR = \"/kaggle/input/asl-signs\"\nEXTEND_TRAIN_DIR = \"/kaggle/input/gislr-extended-train-dataframe\" \n\nprint(\"\\n... BASIC DATA SETUP STARTING ...\\n\")\nprint(\"\\n\\n... LOAD TRAIN DATAFRAME FROM CSV FILE ...\\n\")\n\nLOAD_EXTENDED = True\nif LOAD_EXTENDED and os.path.isfile(os.path.join(EXTEND_TRAIN_DIR, \"extended_train.csv\")):\n train_df = pd.read_csv(os.path.join(EXTEND_TRAIN_DIR, \"extended_train.csv\"))\nelse:\n train_df = pd.read_csv(os.path.join(DATA_DIR, \"train.csv\"))\n train_df[\"path\"] = DATA_DIR+\"/\"+train_df[\"path\"]\ndisplay(train_df)\n\nprint(\"\\n\\n... LOAD SIGN TO PREDICTION INDEX MAP FROM JSON FILE ...\\n\")\ns2p_map = {k.lower():v for k,v in read_json_file(os.path.join(DATA_DIR, \"sign_to_prediction_index_map.json\")).items()}\np2s_map = {v:k for k,v in read_json_file(os.path.join(DATA_DIR, \"sign_to_prediction_index_map.json\")).items()}\nencoder = lambda x: s2p_map.get(x.lower())\ndecoder = lambda x: p2s_map.get(x)\nprint(s2p_map)\n\nDEMO_ROW = 283\nprint(f\"\\n\\n... DEMO SIGN/EVENT DATAFRAME FOR ROW {DEMO_ROW} - SIGN={train_df.iloc[DEMO_ROW]['sign']} ...\\n\")\ndemo_sign_df = get_sign_df(train_df.iloc[DEMO_ROW][\"path\"])\ndisplay(demo_sign_df)\n\n# I messed this function up... will fix later\nplot_event(demo_sign_df)","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:09:51.162321Z","iopub.execute_input":"2023-03-19T12:09:51.162712Z","iopub.status.idle":"2023-03-19T12:10:17.874558Z","shell.execute_reply.started":"2023-03-19T12:09:51.162673Z","shell.execute_reply":"2023-03-19T12:10:17.873426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

5  EXPLORATORY DATA ANALYSIS    ⤒

\n\n
\n\nInitially we will sample approximately 10% of the data to probe, as it is very computationally expensive to open and close all the parquet files. Following my interactive EDA I will switch this percentage to be 100% and allow it to run overnight\n* We will then use the subsampled dataset along with the original to explore the columns and respective parquet files for each isolated sign","metadata":{}},{"cell_type":"markdown","source":"
\n\n

5.0 SUBSAMPLE THE TRAIN DATA

\n\n---\n","metadata":{}},{"cell_type":"code","source":"# During interactive --> 0.001 (0.1%)\n# Save and run-all --> 1.000 (100%)\n\nPCT_TO_EXAMINE = 0.001\nif PCT_TO_EXAMINE < 1.0:\n subsample_train_df = train_df.sample(frac=PCT_TO_EXAMINE, random_state=42).reset_index(drop=True)\nelse:\n subsample_train_df = train_df.copy()\n\n# Remove extra columns to show what we're doing\nsubsample_train_df=subsample_train_df[[\"path\", \"participant_id\", \"sequence_id\", \"sign\"]]\ndisplay(subsample_train_df)","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:10:17.876532Z","iopub.execute_input":"2023-03-19T12:10:17.87683Z","iopub.status.idle":"2023-03-19T12:10:17.899443Z","shell.execute_reply.started":"2023-03-19T12:10:17.876802Z","shell.execute_reply":"2023-03-19T12:10:17.89834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

5.1 EXAMINE THE `PATH` COLUMN

\n\n---\n\nThere's not much here. The path column is simply the path to the landmark file (parquet).\n* Every row and respective filepath is unique\n* Every path is comprised of.\n * The base part of the path --> **`/kaggle/input/asl-signs/train_landmark_files`**\n * The participant_id --> .../16069\n * The sequence_id as the parquet filename --> .../100015657.parquet","metadata":{}},{"cell_type":"code","source":"print(\"\\n... LETS LOOK AT THE PATH COLUMN AND ENSURE ALL PATHS ARE UNIQUE:\")\ndisplay(train_df[\"path\"].describe().to_frame())","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:10:17.901243Z","iopub.execute_input":"2023-03-19T12:10:17.901994Z","iopub.status.idle":"2023-03-19T12:10:17.969264Z","shell.execute_reply.started":"2023-03-19T12:10:17.901957Z","shell.execute_reply":"2023-03-19T12:10:17.968145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

5.2 EXAMINE THE `PARTICIPANT_ID` COLUMN

\n\n---\n\n\n\nIt's also worth pointing out that the folders in the train_landmark_files directory are named based on the participant_id for whom the respective isolated sign event parquet files are for.","metadata":{}},{"cell_type":"code","source":"print(\"\\n... BASICS OF THE PARTICIPANT ID COLUMN:\\n\")\ndisplay(train_df[\"participant_id\"].astype(str).describe().to_frame().T)\n\nprint(\"\\n... WE GET THE COUNT MAP AND GET BASIC STATISTICS:\")\nparticipant_count_map = train_df[\"participant_id\"].value_counts().to_dict()\nprint(\"\\t1. Number of Unique Participants -->\", len(participant_count_map))\nprint(\"\\t2. Average Number of Rows Per Participant -->\", np.array(list(participant_count_map.values())).mean())\nprint(\"\\t3. Standard Deviation in Counts Per Participant -->\", np.array(list(participant_count_map.values())).std())\nprint(\"\\t4. Minimum Number of Examples For One Participant -->\", np.array(list(participant_count_map.values())).min())\nprint(\"\\t5. Maximum Number of Examples For One Participant -->\", np.array(list(participant_count_map.values())).max())\n\nprint(\"\\n\\n... PARTICIPANT ID COLUMN HISTOGRAM:\\n\")\nfig = px.histogram(\n train_df, x=train_df[\"participant_id\"].astype(str), color=\"participant_id\",\n labels={\"x\":\"Participant ID\", \"count\":\"Total Row Count\"}, title=\"Row Counts by Participant ID\",\n category_orders={\"participant_id\": train_df[\"participant_id\"].value_counts().index}\n)\nfig.update_yaxes(title_text=\"Total Row Count\")\nfig.update_layout(showlegend=False)\nfig.show()\n\nprint(\"\\n... GOING FORWARD WE SET THIS COLUMN TO BE A STRING\")\ntrain_df[\"participant_id\"] = train_df[\"participant_id\"].astype(str)\nsubsample_train_df[\"participant_id\"] = subsample_train_df[\"participant_id\"].astype(str)","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:10:17.970954Z","iopub.execute_input":"2023-03-19T12:10:17.971364Z","iopub.status.idle":"2023-03-19T12:10:19.851709Z","shell.execute_reply.started":"2023-03-19T12:10:17.971325Z","shell.execute_reply":"2023-03-19T12:10:19.850591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

5.3 EXAMINE THE `SEQUENCE_ID` COLUMN

\n\n---\n\nThere's not much here. This is a unique value assigned to every isolated sequence/event. One sequence corresponds to a single isolated sign that we have to detect and label.\n* Every value is unique for every row\n","metadata":{}},{"cell_type":"code","source":"print(\"\\n... LETS LOOK AT THE PATH COLUMN AND ENSURE ALL PATHS ARE UNIQUE:\")\ndisplay(train_df[\"sequence_id\"].astype(str).describe().to_frame())\n\nprint(\"\\n... TO CONFIRM... LET'S CHECK HOW MANY PARQUET FILES WE HAVE:\")\nprint(\"\\t--> \", len(glob(os.path.join(DATA_DIR, \"**\", \"**\", \"*.parquet\"))))","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:10:19.853276Z","iopub.execute_input":"2023-03-19T12:10:19.860605Z","iopub.status.idle":"2023-03-19T12:10:25.674857Z","shell.execute_reply.started":"2023-03-19T12:10:19.860571Z","shell.execute_reply":"2023-03-19T12:10:25.67357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

5.4 EXAMINE THE `SIGN` COLUMN

\n\n---\n\nThis is the label for each respective event/sequence.\n \n\n\nIt's a pretty balanced dataset!","metadata":{}},{"cell_type":"code","source":"print(\"\\n... BASICS OF THE PARTICIPANT ID COLUMN:\\n\")\ndisplay(train_df[\"sign\"].describe().to_frame().T)\n\nprint(\"\\n... WE GET THE COUNT MAP AND GET BASIC STATISTICS:\")\nsign_count_map = train_df[\"sign\"].value_counts().to_dict()\nprint(\"\\t1. Number Of Unique Signs -->\", len(sign_count_map))\nprint(\"\\t2. Average Number of Rows Per Sign -->\", np.array(list(sign_count_map.values())).mean())\nprint(\"\\t3. Standard Deviation in Counts Per Sign -->\", np.array(list(sign_count_map.values())).std())\nprint(\"\\t4. Minimum Number of Examples For One Sign -->\", np.array(list(sign_count_map.values())).min())\nprint(\"\\t5. Maximum Number of Examples For One Sign -->\", np.array(list(sign_count_map.values())).max())\n\nprint(\"\\n\\n... SIGN COLUMN HISTOGRAM:\\n\")\nfig = px.histogram(train_df, y=train_df[\"sign\"], color=\"sign\", orientation=\"h\", height=5000,\n labels={\"y\":\"Sign (label)\", \"count\":\"Total Row Count\"}, title=\"Row Counts by Sign (label)\",\n category_orders={\"sign\": train_df[\"sign\"].value_counts().index}\n)\nfig.update_yaxes(title_text=\"Total Row Count\")\nfig.update_layout(showlegend=False)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:10:25.677563Z","iopub.execute_input":"2023-03-19T12:10:25.678356Z","iopub.status.idle":"2023-03-19T12:10:27.036792Z","shell.execute_reply.started":"2023-03-19T12:10:25.678312Z","shell.execute_reply":"2023-03-19T12:10:27.032072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

5.5 INCLUDING SEQUENCE METADATA IN TRAIN DATAFRAME

\n\n---\n\nWe are going to identify certain pieces of relevant metadata that we want to scrape from the parquet files and include in our main dataframe\n\n
\n\nWe will retrieve the following for each sequence\n\n\n
\n\nWhat can we observe about the sequences with this new metadata:\n","metadata":{}},{"cell_type":"code","source":"def get_seq_meta(row, invert_y=True, do_counts=False):\n \"\"\"Calculates and adds metadata to the given row of sign language event data.\n \n Args:\n row (pandas.core.series.Series): A row of sign language event data containing columns:\n path: The file path to the Parquet file containing the landmark data for the event.\n invert_y (bool, optional): Whether to invert the y-coordinate of each landmark. Defaults to True.\n \n Returns:\n pandas.core.series.Series: The input row with added metadata columns:\n start_frame: The frame number of the first frame in the event.\n end_frame: The frame number of the last frame in the event.\n total_frames: The number of frames in the event.\n face_count: The number of landmarks in the 'face' type. [optional]\n pose_count: The number of landmarks in the 'pose' type. [optional]\n left_hand_count: The number of landmarks in the 'left_hand' type. [optional]\n right_hand_count: The number of landmarks in the 'right_hand' type. [optional]\n x_min: The minimum x-coordinate value of any landmark in the event.\n x_max: The maximum x-coordinate value of any landmark in the event.\n y_min: The minimum y-coordinate value of any landmark in the event.\n y_max: The maximum y-coordinate value of any landmark in the event.\n z_min: The minimum z-coordinate value of any landmark in the event.\n z_max: The maximum z-coordinate value of any landmark in the event.\n \"\"\"\n # Extract the sign language event data from the Parquet file at the given path\n df = get_sign_df(row['path'], invert_y=invert_y)\n \n # Count the number of landmarks in each type\n type_counts = df['type'].value_counts(dropna=False).to_dict()\n nan_counts = df.groupby(\"type\")[\"x\"].apply(lambda x: x.isna().sum())\n \n # Calculate metadata for the event and add it to the input row\n row['start_frame'] = df['frame'].min()\n row['end_frame'] = df['frame'].max()\n row['total_frames'] = df['frame'].nunique()\n \n if do_counts:\n for _type in [\"face\", \"pose\", \"left_hand\", \"right_hand\"]:\n row[f'{_type}_count'] = type_counts[_type]\n row[f'{_type}_nan_count'] = nan_counts[_type]\n \n for coord in ['x', 'y', 'z']:\n row[f'{coord}_min'] = df[coord].min()\n row[f'{coord}_max'] = df[coord].max()\n \n return row\n\ntype_kp_map = dict(face=468, left_hand=21, pose=33, right_hand=21)\ncol_order = [\n 'path', 'participant_id', 'sequence_id', 'sign', 'start_frame', 'end_frame', 'total_frames', \n 'face_nan_count', 'face_nan_pct', 'left_hand_nan_count', 'left_hand_nan_pct', 'pose_nan_count', 'pose_nan_pct',\n 'right_hand_nan_count', 'right_hand_nan_pct', 'x_min', 'x_max', 'y_min', 'y_max', 'z_min', 'z_max',\n]\n\nif not LOAD_EXTENDED:\n # Will take around 5-10 minutes on subsample and around 50-100 minutes on the full dataset\n subsample_train_df = subsample_train_df.progress_apply(lambda x: get_seq_meta(x, do_counts=True), axis=1)\n for _type, _count in type_kp_map.items():\n subsample_train_df[f\"{_type}_appears_pct\"] = subsample_train_df[f\"{_type}_count\"]/(subsample_train_df[f\"total_frames\"]*_count)\n subsample_train_df[f\"{_type}_nan_pct\"] = subsample_train_df[f\"{_type}_nan_count\"]/(subsample_train_df[f\"total_frames\"]*_count)\n # Extended save for later...\n subsample_train_df.to_csv(\"extended_train.csv\", index=False)\n display(subsample_train_df)\nelse:\n del subsample_train_df\n for _type, _count in type_kp_map.items():\n train_df[f\"{_type}_appears_pct\"] = train_df[f\"{_type}_count\"]/(train_df[f\"total_frames\"]*_count)\n train_df[f\"{_type}_nan_pct\"] = train_df[f\"{_type}_nan_count\"]/(train_df[f\"total_frames\"]*_count)\n train_df = train_df[col_order]\n display(train_df)","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:10:27.046578Z","iopub.execute_input":"2023-03-19T12:10:27.047374Z","iopub.status.idle":"2023-03-19T12:10:27.229162Z","shell.execute_reply.started":"2023-03-19T12:10:27.047317Z","shell.execute_reply":"2023-03-19T12:10:27.227113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following plot shows the general distributions (independently plotted).\n* Note the Y axis is log scale.\n* 0.0 (Left Side) - Indicates that no values are missing\n* 1.0 (Right Side) - Indicates that all values are missing\n\n
\n\nQuick Takeaways\n* Face points can be NaN although it is less common than in the Hand data\n* Pose points are never NaN\n* Left and Right hand distributions are similar but Right Hand is full NaN less than Left Hand\n* Pose, Left-Hand, and Right-Hand all have intermediate (not all missing or all present) sequences, however, they are less common than the case where all points are NaN or valid.","metadata":{}},{"cell_type":"code","source":"def title_map_fn(ann):\n title_map = {\n 'face_nan_pct': 'Percentage Of Face Data Points That Are NaN', \n 'left_hand_nan_pct': 'Percentage Of Left Hand Data Points That Are NaN',\n 'pose_nan_pct': 'Percentage Of Pose Data Points That Are NaN',\n 'right_hand_nan_pct': 'Percentage Of Right Hand Data Points That Are NaN'}\n ann.text = title_map.get(ann.text[1:])\n \nfig = px.histogram(train_df, [\"face_nan_pct\", \"left_hand_nan_pct\", \"pose_nan_pct\", \"right_hand_nan_pct\"], height=750,\n labels={'variable': '', 'count': 'Frequency (LOG)', 'value':\"Percentage of Points That Are NaN\"}, log_y=True, facet_col='variable', nbins=20, opacity=0.75,\n facet_col_wrap=2, facet_col_spacing=0.05)\nfig.update_yaxes(title_text='Frequency (LOG)', col=1)\nfig.for_each_annotation(title_map_fn)\nfig.update_layout(showlegend=False)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:10:27.235416Z","iopub.execute_input":"2023-03-19T12:10:27.236221Z","iopub.status.idle":"2023-03-19T12:10:27.704314Z","shell.execute_reply.started":"2023-03-19T12:10:27.236169Z","shell.execute_reply":"2023-03-19T12:10:27.703288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Check that all the frames in all the files have the same ordering\n# for _path in tqdm(train_df.sample(n=500, random_state=42).reset_index(drop=True)[\"path\"].values, total=500):\n# for _frame_types in get_sign_df(_path).groupby(\"frame\")[\"type\"].apply(list).values:\n# current_idx = 0\n# face_check = _frame_types[current_idx:current_idx+type_kp_map[\"face\"]].count(\"face\")==type_kp_map[\"face\"]\n# current_idx+=type_kp_map[\"face\"]\n# if not face_check:\n# print(\"face\")\n# raise ValueError()\n# lh_check = _frame_types[current_idx:current_idx+type_kp_map[\"left_hand\"]].count(\"left_hand\")==type_kp_map[\"left_hand\"]\n# current_idx+=type_kp_map[\"left_hand\"]\n# if not lh_check:\n# print(\"lh\")\n# raise ValueError()\n# pose_check = _frame_types[current_idx:current_idx+type_kp_map[\"pose\"]].count(\"pose\")==type_kp_map[\"pose\"]\n# current_idx+=type_kp_map[\"pose\"]\n# if not pose_check:\n# print(\"pose\")\n# raise ValueError()\n# rh_check = _frame_types[current_idx:current_idx+type_kp_map[\"right_hand\"]].count(\"right_hand\")==type_kp_map[\"right_hand\"]\n# if not rh_check:\n# print(\"rh\")\n# raise ValueError()\n\n# Landmark IDs start at 0 for each respective type and count up\nFRAME_TYPE_ORDER_DETAIL = demo_sign_df.groupby(\"frame\")[\"type\"].apply(list).values[0]\nFRAME_TYPE_ORDER = sorted(set(FRAME_TYPE_ORDER_DETAIL))\nprint(FRAME_TYPE_ORDER)\nprint(type_kp_map)\n\nFRAME_TYPE_IDX_MAP = {\n \"face\" : np.arange(0, 468),\n \"left_hand\" : np.arange(468, 489),\n \"pose\" : np.arange(489, 522),\n \"right_hand\" : np.arange(522, 543),\n}\nfor k,v in FRAME_TYPE_IDX_MAP.items():\n print(k, FRAME_TYPE_ORDER_DETAIL[v[0]:v[1]].count(k)==(v[1]-v[0]))","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:10:27.705701Z","iopub.execute_input":"2023-03-19T12:10:27.706303Z","iopub.status.idle":"2023-03-19T12:10:27.722783Z","shell.execute_reply.started":"2023-03-19T12:10:27.706264Z","shell.execute_reply":"2023-03-19T12:10:27.72134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

5.6 OUTLIER ANALYSIS

\n\n---\n\nIn this section we are going to look to see if we can detect outliers for any of the relevant variables\n\nTHE FOLLOWING COLUMNS/VARS DO NOT HAVE SIGNIFICANT OUTLIERS\n\n\nTHE FOLLOWING ANALYSIS PERTAINS TO FRAME INFORMATION AND SEQUENCE LENGTH\n\n","metadata":{}},{"cell_type":"code","source":"# Create a box plot\nfig = px.box(train_df, y=['start_frame', 'end_frame', 'total_frames'],\n title='Box Plot of Start Frame, End Frame, and Total Frames')\n\n# Customize the box and whisker colors and width\nfig.update_traces(boxmean=True)\n\n# Customize the x and y axis labels\nfig.update_xaxes(title_text='Frame Measure')\nfig.update_yaxes(title_text='Number of Frames')\n\n# Show the plot\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:10:27.72426Z","iopub.execute_input":"2023-03-19T12:10:27.725523Z","iopub.status.idle":"2023-03-19T12:10:28.797132Z","shell.execute_reply.started":"2023-03-19T12:10:27.72547Z","shell.execute_reply":"2023-03-19T12:10:28.795532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"long_seqs = train_df[train_df.total_frames>train_df.total_frames.quantile(0.9)]\nnotlong_seqs = train_df[train_df.total_frames<=train_df.total_frames.quantile(0.9)]\n\nlong_seq_distribution = {k:v/len(long_seqs) for k,v in long_seqs.sign.value_counts().items()}\nnotlong_seq_distribution = {k:v/len(notlong_seqs) for k,v in notlong_seqs.sign.value_counts().items()}\nprint(\"\\n... COMPARE DISTRIBUTIONS:\")\nfor i, k in enumerate(sorted(train_df.sign.unique())):\n if i==0: print(f\"\\n\\t{'SIGN':<15} --> LONG vs. OTHER \\n{'-'*50}\")\n print(f\"\\t{k:<15} --> {long_seq_distribution.get(k, 0.0):.4f} vs. {notlong_seq_distribution.get(k, 0.0):.4f}\")","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:10:28.800909Z","iopub.execute_input":"2023-03-19T12:10:28.801454Z","iopub.status.idle":"2023-03-19T12:10:28.861123Z","shell.execute_reply.started":"2023-03-19T12:10:28.80138Z","shell.execute_reply":"2023-03-19T12:10:28.860048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.sort_values(by=\"total_frames\", ascending=False).head(1)\nlong_sign_df = get_sign_df(train_df.sort_values(by=\"total_frames\", ascending=False).path.values[0])\nplot_event(long_sign_df, style=\"hands\")","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:10:28.86257Z","iopub.execute_input":"2023-03-19T12:10:28.863677Z","iopub.status.idle":"2023-03-19T12:11:42.512538Z","shell.execute_reply.started":"2023-03-19T12:10:28.863633Z","shell.execute_reply":"2023-03-19T12:11:42.511426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

6  BASELINE    ⤒

\n\nRemember that the hosts will be using the load_relevant_data_subset function which only yields x,y,z values. We will keep this in mind while building our models.\n\n---\n\nThe Dataset\n* I created tfrecords in a separate notebook:\n * TFRecord Creation Notebook\n* These tfrecords contain examples with the following structure:\n * 3 Key Frames for Each Type (Usually the first and then 2 most disparate non-zero frames)\n * Only X,Y coordinates\n * Each type is stored separately – 'face': 468, 'left_hand': 21, 'pose': 33, 'right_hand': 21\n * The label is stored with them as an integer (sparse)\n\n---\n\nVERY WIP –– I'M HAVING SOME ISSUES BUT I'LL CLEAN THIS UP SOON

For now I'll just make a simple model with the basic mean np data people are using...\n\n---\n\nThe Model\n* We are only going to use hands to start so we discard pose and face.\n * We know only one hand is present, and we may be able to leverage this in the future (i.e. break out into two models and only call the one we care about)\n* We are going to have two batch dimensions\n * The real batch dimension\n * The frame batch dimension\n * This will allow us to process/learn more and we can batch it similarly during inference.","metadata":{}},{"cell_type":"code","source":"train_x = np.load(\"/kaggle/input/gislr-feature-data/feature_data.npy\").astype(np.float32)\ntrain_y = np.load(\"/kaggle/input/gislr-feature-data/feature_labels.npy\").astype(np.uint8)\nBATCH_SIZE = 64\n\nN_TOTAL = train_x.shape[0]\nVAL_PCT = 0.1\nN_VAL = int(N_TOTAL*VAL_PCT)\nN_TRAIN = N_TOTAL-N_VAL\n\nrandom_idxs = random.sample(range(N_TOTAL), N_TOTAL)\ntrain_idxs, val_idxs = np.array(random_idxs[:N_TRAIN]), np.array(random_idxs[N_TRAIN:])\n\nval_x, val_y = train_x[val_idxs], train_y[val_idxs]\ntrain_x, train_y = train_x[train_idxs], train_y[train_idxs]\n\n# train_ds = tf.data.Dataset.from_tensor_slices((train_x, train_y))\\\n# .shuffle(N_TRAIN)\\\n# .batch(BATCH_SIZE, drop_remainder=True)\\\n# .prefetch(tf.data.AUTOTUNE)\n\n# val_ds = tf.data.Dataset.from_tensor_slices((train_x, train_y))\\\n# .shuffle(N_VAL)\\\n# .batch(BATCH_SIZE, drop_remainder=True)\\\n# .prefetch(tf.data.AUTOTUNE)\n\n# train_ds, val_ds","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:11:42.514808Z","iopub.execute_input":"2023-03-19T12:11:42.515699Z","iopub.status.idle":"2023-03-19T12:12:13.056874Z","shell.execute_reply.started":"2023-03-19T12:11:42.515653Z","shell.execute_reply":"2023-03-19T12:12:13.05567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def fc_block(inputs, output_channels, dropout=0.2):\n x = tf.keras.layers.Dense(output_channels)(inputs)\n x = tf.keras.layers.BatchNormalization()(x)\n x = tf.keras.layers.Activation(\"gelu\")(x)\n x = tf.keras.layers.Dropout(dropout)(x)\n return x\n\ndef get_model(n_labels=250, init_fc=512, n_blocks=2, _dropout_1=0.2, _dropout_2=0.6, flat_frame_len=3258):\n _inputs = tf.keras.layers.Input(shape=(flat_frame_len,))\n x = _inputs\n \n # Define layers\n for i in range(n_blocks):\n x = fc_block(\n x, output_channels=init_fc//(2**i), \n dropout=_dropout_1 if (1+i)!=n_blocks else _dropout_2\n )\n \n # Define output layer\n _outputs = tf.keras.layers.Dense(n_labels, activation=\"softmax\")(x)\n \n # Build the model\n model = tf.keras.models.Model(inputs=_inputs, outputs=_outputs)\n return model\n\nmodel = get_model()\nmodel.compile(tf.keras.optimizers.Adam(0.000333), \"sparse_categorical_crossentropy\", metrics=\"acc\")\nmodel.summary()\n\ntf.keras.utils.plot_model(model)","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:12:13.058216Z","iopub.execute_input":"2023-03-19T12:12:13.058629Z","iopub.status.idle":"2023-03-19T12:12:17.290688Z","shell.execute_reply.started":"2023-03-19T12:12:13.058587Z","shell.execute_reply":"2023-03-19T12:12:17.289349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!mkdir models\ncb_list = [\n tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True, verbose=1),\n tf.keras.callbacks.ReduceLROnPlateau(patience=2, factor=0.8, verbose=1)\n]\nhistory = model.fit(train_x, train_y, validation_data=(val_x, val_y), epochs=100, callbacks=cb_list, batch_size=BATCH_SIZE)\nmodel.save(\"./models/asl_model\")","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:12:17.295665Z","iopub.execute_input":"2023-03-19T12:12:17.298053Z","iopub.status.idle":"2023-03-19T12:19:20.583652Z","shell.execute_reply.started":"2023-03-19T12:12:17.298009Z","shell.execute_reply":"2023-03-19T12:19:20.582459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.evaluate(val_x, val_y)\nfor x,y in zip(val_x[:10], val_y[:10]):\n print(f\"PRED: {decoder(np.argmax(model.predict(tf.expand_dims(x, axis=0), verbose=0), axis=-1)[0]):<20} – GT: {decoder(y)}\")","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:19:20.592384Z","iopub.execute_input":"2023-03-19T12:19:20.595223Z","iopub.status.idle":"2023-03-19T12:19:22.472048Z","shell.execute_reply.started":"2023-03-19T12:19:20.595161Z","shell.execute_reply":"2023-03-19T12:19:22.471059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class PrepInputs(tf.keras.layers.Layer):\n def __init__(self, face_idx_range=(0, 468), lh_idx_range=(468, 489), \n pose_idx_range=(489, 522), rh_idx_range=(522, 543)):\n super(PrepInputs, self).__init__()\n self.idx_ranges = [face_idx_range, lh_idx_range, pose_idx_range, rh_idx_range]\n self.flat_feat_lens = [3*(_range[1]-_range[0]) for _range in self.idx_ranges]\n \n def call(self, x_in):\n \n # Split the single vector into 4\n xs = [x_in[:, _range[0]:_range[1], :] for _range in self.idx_ranges]\n \n # Reshape based on specific number of keypoints\n xs = [tf.reshape(_x, (-1, flat_feat_len)) for _x, flat_feat_len in zip(xs, self.flat_feat_lens)]\n \n # Drop empty rows - Empty rows are present in \n # --> pose, lh, rh\n # --> so we don't have to for face\n xs[1:] = [\n tf.boolean_mask(_x, tf.reduce_all(tf.logical_not(tf.math.is_nan(_x)), axis=1), axis=0)\n for _x in xs[1:]\n ]\n \n # Get means and stds\n x_means = [tf.math.reduce_mean(_x, axis=0) for _x in xs]\n x_stds = [tf.math.reduce_std(_x, axis=0) for _x in xs]\n \n x_out = tf.concat([*x_means, *x_stds], axis=0)\n x_out = tf.where(tf.math.is_finite(x_out), x_out, tf.zeros_like(x_out))\n return tf.expand_dims(x_out, axis=0)\n \nPrepInputs()(load_relevant_data_subset(train_df.path[0]))","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:19:22.47387Z","iopub.execute_input":"2023-03-19T12:19:22.474264Z","iopub.status.idle":"2023-03-19T12:19:22.574051Z","shell.execute_reply.started":"2023-03-19T12:19:22.474227Z","shell.execute_reply":"2023-03-19T12:19:22.572642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\n\nclass TFLiteModel(tf.Module):\n \"\"\"\n TensorFlow Lite model that takes input tensors and applies:\n – a preprocessing model\n – the ISLR model \n \"\"\"\n\n def __init__(self, islr_model):\n \"\"\"\n Initializes the TFLiteModel with the specified preprocessing model and ISLR model.\n \"\"\"\n super(TFLiteModel, self).__init__()\n\n # Load the feature generation and main models\n self.prep_inputs = PrepInputs()\n self.islr_model = islr_model\n \n @tf.function(input_signature=[tf.TensorSpec(shape=[None, 543, 3], dtype=tf.float32, name='inputs')])\n def __call__(self, inputs):\n \"\"\"\n Applies the feature generation model and main model to the input tensors.\n\n Args:\n inputs: Input tensor with shape [batch_size, 543, 3].\n\n Returns:\n A dictionary with a single key 'outputs' and corresponding output tensor.\n \"\"\"\n x = self.prep_inputs(tf.cast(inputs, dtype=tf.float32))\n outputs = self.islr_model(x)[0, :]\n\n # Return a dictionary with the output tensor\n return {'outputs': outputs}\n\ntflite_keras_model = TFLiteModel(islr_model=model)\ndemo_output = tflite_keras_model(load_relevant_data_subset(train_df.path[0]))[\"outputs\"]\ndecoder(np.argmax(demo_output.numpy(), axis=-1))","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:19:22.57596Z","iopub.execute_input":"2023-03-19T12:19:22.576421Z","iopub.status.idle":"2023-03-19T12:19:22.979439Z","shell.execute_reply.started":"2023-03-19T12:19:22.576356Z","shell.execute_reply":"2023-03-19T12:19:22.978376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"keras_model_converter = tf.lite.TFLiteConverter.from_keras_model(tflite_keras_model)\ntflite_model = keras_model_converter.convert()\nwith open('/kaggle/working/models/model.tflite', 'wb') as f:\n f.write(tflite_model)\n!zip submission.zip /kaggle/working/models/model.tflite\n\n!pip install tflite-runtime\nimport tflite_runtime.interpreter as tflite\n\ninterpreter = tflite.Interpreter(\"/kaggle/working/models/model.tflite\")\nfound_signatures = list(interpreter.get_signature_list().keys())\n# if REQUIRED_SIGNATURE not in found_signatures:\n# raise KernelEvalException('Required input signature not found.')\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\n\noutput = prediction_fn(inputs=load_relevant_data_subset(train_df.path[0]))\nsign = np.argmax(output[\"outputs\"])\n\nprint(\"PRED : \", decoder(sign))\nprint(\"GT : \", train_df.sign[0])","metadata":{"execution":{"iopub.status.busy":"2023-03-19T12:19:22.980959Z","iopub.execute_input":"2023-03-19T12:19:22.981299Z","iopub.status.idle":"2023-03-19T12:19:37.803923Z","shell.execute_reply.started":"2023-03-19T12:19:22.981263Z","shell.execute_reply":"2023-03-19T12:19:37.802489Z"},"trusted":true},"execution_count":null,"outputs":[]}]}