{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **American Sign Language Recognition**","metadata":{}},{"cell_type":"markdown","source":"#### Goal - Classify isolated American Sign Language (ASL) signs by creating a TensorFlow Lite model trained on labeled landmark data extracted using the MediaPipe Holistic Solution.","metadata":{}},{"cell_type":"markdown","source":"We have x-y-z coordinates of landmark indices of hand,face and pose for each frame of a sequence. We need to use all/some of the frames to classify the sequence as a whole into the 250 odd signs there are. There are a total of 21 participants in training data and close to ~4500 sequences per person. Below are the landmark indices of hand from the mediapipe page. There's no public test data in this competition.","metadata":{}},{"cell_type":"markdown","source":"![](https://mediapipe.dev/images/mobile/hand_landmarks.png)","metadata":{}},{"cell_type":"markdown","source":"\n<img src=\"https://mediapipe.dev/images/mobile/holistic_sports_and_gestures_example.gif\">","metadata":{}},{"cell_type":"code","source":"import os\nimport glob\nimport tqdm\nimport random\nimport json\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n# Visualization Imports (overkill)\nfrom matplotlib.animation import FuncAnimation\nfrom matplotlib.colors import ListedColormap\nfrom matplotlib.patches import Rectangle\nimport matplotlib.patches as patches\nimport plotly.graph_objects as go\nfrom IPython.display import HTML\nfrom tqdm.notebook import tqdm; tqdm.pandas();\nimport plotly.express as px\n\nLANDMARK_FILES_DIR = \"/kaggle/input/asl-signs/train_landmark_files\"\nTRAIN_FILE = \"/kaggle/input/asl-signs/train.csv\"\nPREDICTION_INDEX_FILE = \"/kaggle/input/asl-signs/sign_to_prediction_index_map.json\"","metadata":{"execution":{"iopub.status.busy":"2023-03-27T22:40:30.366918Z","iopub.execute_input":"2023-03-27T22:40:30.367352Z","iopub.status.idle":"2023-03-27T22:40:30.381753Z","shell.execute_reply.started":"2023-03-27T22:40:30.367310Z","shell.execute_reply":"2023-03-27T22:40:30.380355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"participants = os.listdir(LANDMARK_FILES_DIR)\nprint(f\"Total number of participants = {len(participants)}\")\nprint(f\"Average number of sequences per participant = {len(glob.glob(LANDMARK_FILES_DIR + '/*/*.parquet'))/len(participants)}\")","metadata":{"execution":{"iopub.status.busy":"2023-03-27T19:21:02.237499Z","iopub.execute_input":"2023-03-27T19:21:02.238959Z","iopub.status.idle":"2023-03-27T19:21:05.337652Z","shell.execute_reply.started":"2023-03-27T19:21:02.238901Z","shell.execute_reply":"2023-03-27T19:21:05.336295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_idx = json.loads(open(PREDICTION_INDEX_FILE,'r').read())\nprint(len(pred_idx))","metadata":{"execution":{"iopub.status.busy":"2023-03-27T19:21:06.411554Z","iopub.execute_input":"2023-03-27T19:21:06.412680Z","iopub.status.idle":"2023-03-27T19:21:06.421087Z","shell.execute_reply.started":"2023-03-27T19:21:06.412612Z","shell.execute_reply":"2023-03-27T19:21:06.420150Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have 250 different signs and associated frames.","metadata":{}},{"cell_type":"code","source":"sample = pd.read_parquet(\"/kaggle/input/asl-signs/train_landmark_files/53618/1001896056.parquet\")\nprint(f\"Sample shape = {sample.shape}\")\nsample.sample(10)","metadata":{"execution":{"iopub.status.busy":"2023-03-27T19:59:00.636342Z","iopub.execute_input":"2023-03-27T19:59:00.637762Z","iopub.status.idle":"2023-03-27T19:59:00.672856Z","shell.execute_reply.started":"2023-03-27T19:59:00.637706Z","shell.execute_reply":"2023-03-27T19:59:00.671368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"All different types of landmark = {sample.type.unique()}\")\nframes = sorted(sample.frame.unique())\nprint(frames)","metadata":{"execution":{"iopub.status.busy":"2023-03-27T19:59:03.710811Z","iopub.execute_input":"2023-03-27T19:59:03.711247Z","iopub.status.idle":"2023-03-27T19:59:03.719666Z","shell.execute_reply.started":"2023-03-27T19:59:03.711208Z","shell.execute_reply":"2023-03-27T19:59:03.718140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"frame_24 = sample.loc[sample['frame'] == 24]\nframe_24","metadata":{"execution":{"iopub.status.busy":"2023-03-27T19:59:07.900332Z","iopub.execute_input":"2023-03-27T19:59:07.900764Z","iopub.status.idle":"2023-03-27T19:59:07.923786Z","shell.execute_reply.started":"2023-03-27T19:59:07.900713Z","shell.execute_reply":"2023-03-27T19:59:07.922443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"frame_24_right_hand = frame_24.loc[frame_24['type'] == \"right_hand\"]\nframe_24_right_hand","metadata":{"execution":{"iopub.status.busy":"2023-03-27T20:00:21.534617Z","iopub.execute_input":"2023-03-27T20:00:21.535041Z","iopub.status.idle":"2023-03-27T20:00:21.555346Z","shell.execute_reply.started":"2023-03-27T20:00:21.535004Z","shell.execute_reply":"2023-03-27T20:00:21.553811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"edges = [(0,1),(1,2),(2,3),(3,4),(0,5),(0,17),(5,6),(6,7),(7,8),(5,9),(9,10),(10,11),(11,12),\n         (9,13),(13,14),(14,15),(15,16),(13,17),(17,18),(18,19),(19,20)]\n\nx = list(frame_24_right_hand.x)\ny = list(frame_24_right_hand.y)\n\nplt.scatter(frame_24_right_hand.x, frame_24_right_hand.y, color='dodgerblue')\nfor i in range(len(x)):\n    plt.text(x[i], y[i], str(i))\n\nfor edge in edges:\n    plt.plot([x[edge[0]], x[edge[1]]], [y[edge[0]], y[edge[1]]], color='salmon')","metadata":{"execution":{"iopub.status.busy":"2023-03-27T20:03:30.002181Z","iopub.execute_input":"2023-03-27T20:03:30.002590Z","iopub.status.idle":"2023-03-27T20:03:30.318420Z","shell.execute_reply.started":"2023-03-27T20:03:30.002553Z","shell.execute_reply":"2023-03-27T20:03:30.317028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Train Data**","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(TRAIN_FILE)\nprint(f\"Train shape = {train.shape}\")\ntrain.sample(10)","metadata":{"execution":{"iopub.status.busy":"2023-03-27T20:34:53.837672Z","iopub.execute_input":"2023-03-27T20:34:53.838135Z","iopub.status.idle":"2023-03-27T20:34:54.062351Z","shell.execute_reply.started":"2023-03-27T20:34:53.838095Z","shell.execute_reply":"2023-03-27T20:34:54.061474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.groupby(['participant_id']).agg(unique_signs=('sign', 'nunique')).reset_index()","metadata":{"execution":{"iopub.status.busy":"2023-03-27T20:35:32.855055Z","iopub.execute_input":"2023-03-27T20:35:32.856058Z","iopub.status.idle":"2023-03-27T20:35:32.907835Z","shell.execute_reply.started":"2023-03-27T20:35:32.856003Z","shell.execute_reply":"2023-03-27T20:35:32.906395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"count_per_sign = train.groupby(['sign'])['sequence_id'].count().reset_index().sort_values(['sequence_id'], ascending=False)\nfig, ax = plt.subplots(1, 2, figsize=(16,5))\nax[0].bar(range(10), count_per_sign['sequence_id'][:10], color='salmon')\nax[1].bar(range(10), count_per_sign['sequence_id'][-10:], color='dodgerblue')\n\nax[0].set_xticks(range(10))\nax[1].set_xticks(range(10))\n\nax[0].set_xticklabels(count_per_sign['sign'][:10])\nax[1].set_xticklabels(count_per_sign['sign'][-10:])\n\nax[0].set_ylim(250,420)\nax[1].set_ylim(250,420)\n\nax[0].set_xlabel(\"Most frequent signs\")\nax[1].set_xlabel(\"Least frequent signs\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-27T20:36:13.158793Z","iopub.execute_input":"2023-03-27T20:36:13.159230Z","iopub.status.idle":"2023-03-27T20:36:13.561002Z","shell.execute_reply.started":"2023-03-27T20:36:13.159190Z","shell.execute_reply":"2023-03-27T20:36:13.559868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Details of Sign Sequence Data**\nCombining the train and sequence landmarks data,we take a look at how different hands are used in a sign or how elaborate (long) they are. It seems like in general left hand is used slightly more than the right hand and use of both hands simultaneously is very rare.","metadata":{}},{"cell_type":"code","source":"def get_details_per_sign(sign):\n    train_sign_sample = train[train['sign'] == sign]\n    n_frames = 0\n    n_left_hand = 0\n    n_right_hand = 0\n    n_face = 0\n    n_both_hands = 0\n    for _,row in train_sign_sample.iterrows():\n        df = pd.read_parquet(os.path.join(\"/kaggle/input/asl-signs\", row.path))\n        n_frames += df['frame'].nunique()\n        n_left_hand += np.sum(df[(df['type'] == 'left_hand') & (df['landmark_index'] == 0)]['x'].isnull() == False)\n        n_right_hand += np.sum(df[(df['type'] == 'right_hand') & (df['landmark_index'] == 0)]['x'].isnull() == False)\n        n_face += np.sum(df[(df['type'] == 'face') & (df['landmark_index'] == 0)]['x'].isnull() == False)\n        \n        df_both_hands = df[(df['type'] == 'left_hand') & (df['landmark_index'] == 0)].merge(\\\n                            df[(df['type'] == 'right_hand') & (df['landmark_index'] == 0)], on='frame', suffixes=('_left', '_right'))\n        n_both_hands += df_both_hands[(df_both_hands['x_left'].isnull() == False) &\\\n                                             (df_both_hands['x_right'].isnull() == False)]['frame'].count()\n            \n    return n_frames/len(train_sign_sample), n_left_hand/n_frames, n_right_hand/n_frames, n_both_hands/n_frames, n_face/n_frames\n\n\nfor sign in ['cloud', 'thankyou', 'donkey', 'because', 'yellow', 'icecream']:\n    total_frames, pct_left, pct_right, pct_both, pct_face = get_details_per_sign(sign)\n    print(\"=\"*20, f\"{sign}\", \"=\"*20)\n    print(f\"Average Number of Frames per Sequence = {total_frames}\")\n    print(f\"Percent of Frames in which a body part exists: Left Hand: {pct_left*100:.02f} %, Right Hand: {pct_right*100:.02f} %, Both Hands: {pct_both*100:.02f} %, Face: {pct_face*100:.02f} %\")\n    print()","metadata":{"execution":{"iopub.status.busy":"2023-03-27T20:38:19.400574Z","iopub.execute_input":"2023-03-27T20:38:19.401001Z","iopub.status.idle":"2023-03-27T20:39:59.506749Z","shell.execute_reply.started":"2023-03-27T20:38:19.400965Z","shell.execute_reply":"2023-03-27T20:39:59.504564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***An important point to note here is that according to details mentioned on the kaggle competition the participants in the data are supposed to have used only one hand and that the signs in the data are doable with a single hand***","metadata":{}},{"cell_type":"markdown","source":"# **Visualization Helper Functions**","metadata":{}},{"cell_type":"code","source":"def get_sign_df(pq_path, invert_y=True):\n    sign_df = pd.read_parquet(\"/kaggle/input/asl-signs/\"+pq_path)\n    \n    # y value is inverted (Thanks @danielpeshkov)\n    if invert_y: sign_df[\"y\"] *= -1 \n        \n    return sign_df\n\ndef get_hand_points(hand):\n    \"\"\"Return x, y lists of normalized spatial coordinates for each finger in the hand dataframe.\"\"\"\n    def __get_hand_ax(_axis):\n        return [np.nan_to_num(_x) for _x in \n            [hand.iloc[i][_axis] for i in range(5)]+\\\n            [[hand.iloc[i][_axis] for i in range(j, j+4)] for j in range(5, 21, 4)]+\\\n            [hand.iloc[i][_axis] for i in special_pts]]\n    special_pts = [0, 5, 9, 13, 17, 0]\n    return [__get_hand_ax(_ax) for _ax in ['x','y','z']]\n\ndef get_pose_points(pose):\n    \"\"\"\n    Extracts x and y coordinates from the provided dataframe for pose landmarks.\n\n    Args:\n        pose (pandas.DataFrame): Dataframe containing pose landmarks with columns ['x', 'y', 'z', 'visibility', 'presence'].\n\n    Returns:\n        tuple: Two lists of x and y coordinates, respectively.\n\n    \"\"\"\n    def __get_pose_ax(_axis):\n        return [np.nan_to_num(_x) for _x in [\n            [pose.iloc[i][_axis] for i in [8, 6, 5, 4, 0, 1, 2, 3, 7]], \n            [pose.iloc[i][_axis] for i in [10, 9]], \n            [pose.iloc[i][_axis] for i in [22, 16, 20, 18, 16, 14, 12, 11, 13, 15, 17, 19, 15, 21]], \n            [pose.iloc[i][_axis] for i in [12, 24, 26, 28, 30, 32, 28]], \n            [pose.iloc[i][_axis] for i in [11, 23, 25, 27, 29, 31, 27]], \n            [pose.iloc[i][_axis] for i in [24, 23]]\n        ]]\n    return [__get_pose_ax(_ax) for _ax in ['x','y','z']]\n\n\ndef animation_frame(f, event_df, ax, ax_pad=0.2, style=\"full\", \n                    face_color=\"spring\", pose_color=\"autumn\", lh_color=\"winter\", rh_color=\"summer\"):\n    \"\"\"\n    Function called by FuncAnimation to animate the plot with the provided frame.\n\n    Args:\n        f (int): The current frame number.\n\n    Returns:\n        None.\n    \"\"\"\n    \n    face_color = plt.cm.get_cmap(face_color)\n    pose_color = plt.cm.get_cmap(pose_color)\n    rh_color = plt.cm.get_cmap(rh_color)\n    lh_color = plt.cm.get_cmap(lh_color)\n    \n    sign_df = event_df.copy()\n    \n    # Clear axis and fix the axis\n    ax.clear()\n    if style==\"full\":\n        xmin = sign_df['x'].min() - ax_pad\n        xmax = sign_df['x'].max() + ax_pad\n        ymin = sign_df['y'].min() - ax_pad\n        ymax = sign_df['y'].max() + ax_pad\n    elif style==\"hands\":\n        xmin = sign_df[sign_df.type.isin([\"left_hand\", \"right_hand\"])]['x'].min() - ax_pad\n        xmax = sign_df[sign_df.type.isin([\"left_hand\", \"right_hand\"])]['x'].max() + ax_pad\n        ymin = sign_df[sign_df.type.isin([\"left_hand\", \"right_hand\"])]['y'].min() - ax_pad\n        ymax = sign_df[sign_df.type.isin([\"left_hand\", \"right_hand\"])]['y'].max() + ax_pad\n    else:\n        xmin = sign_df[sign_df.type==style]['x'].min() - ax_pad\n        xmax = sign_df[sign_df.type==style]['x'].max() + ax_pad\n        ymin = sign_df[sign_df.type==style]['y'].min() - ax_pad\n        ymax = sign_df[sign_df.type==style]['y'].max() + ax_pad\n    \n    ax.set_xlim(xmin, xmax)\n    ax.set_ylim(ymin, ymax)\n    ax.axis(False) # Remove the axis lines\n    \n    # Normalize depth\n    zmin, zmax = sign_df['z'].min(), sign_df['z'].max()\n    sign_df['z'] = (sign_df['z']-zmin)/(zmax-zmin)\n    \n    # Get data for current frame\n    frame = sign_df[sign_df.frame==f]\n    \n    # Left Hand\n    if style.lower() in [\"left_hand\", \"hands\", \"full\"]:\n        left = frame[frame.type=='left_hand']\n        lx, ly, lz = get_hand_points(left)\n        for i in range(len(lx)):\n            if type(lx[i])!=np.float64:\n                lh_clr = [lh_color(((np.abs(_x)+np.abs(_y))/2)) for _x, _y in zip(lx[i], ly[i])]\n                lh_clr = tuple(sum(_x)/len(_x) for _x in zip(*lh_clr))\n            else: \n                lh_clr = lh_color(((np.abs(lx[i])+np.abs(ly[i]))/2))\n            ax.plot(lx[i], ly[i], color=lh_clr, alpha=lz[i].mean())\n    \n    # Right Hand\n    if style.lower() in [\"right_hand\", \"hands\", \"full\"]:\n        right = frame[frame.type=='right_hand']\n        rx, ry, rz = get_hand_points(right)\n        for i in range(len(rx)):\n            if type(rx[i])!=np.float64:\n                rh_clr = [rh_color((np.abs(_x)+np.abs(_y))/2) for _x, _y in zip(rx[i], ry[i])] \n                rh_clr = tuple(sum(_x)/len(_x) for _x in zip(*rh_clr))\n            else:\n                rh_clr = rh_color(((np.abs(rx[i])+np.abs(ry[i]))/2))\n            ax.plot(rx[i], ry[i], color=rh_clr, alpha=rz[i].mean())\n    \n    # Pose\n    if style.lower() in [\"pose\", \"full\"]:\n        pose = frame[frame.type=='pose']\n        px, py, pz = get_pose_points(pose)\n        for i in range(len(px)):\n            if type(px[i])!=np.float64:\n                pose_clr = [pose_color(((np.abs(_x)+np.abs(_y))/2)) for _x, _y in zip(px[i], py[i])]\n                pose_clr = tuple(sum(_x)/len(_x) for _x in zip(*pose_clr))\n            else: \n                pose_clr = pose_color(((np.abs(px[i])+np.abs(py[i]))/2))\n            ax.plot(px[i], py[i], color=pose_clr, alpha=pz[i].mean())\n        \n    if style.lower() in [\"face\", \"full\"]:\n        face = frame[frame.type=='face'][['x', 'y', 'z']].values\n        fx, fy, fz = face[:,0], face[:,1], face[:,2]\n        for i in range(len(fx)):\n            ax.plot(fx[i], fy[i], '.', color=pose_color(fz[i]), alpha=fz[i])\n    \n    # Use this so we don't get an extra return\n    plt.close()\n    \n    \ndef plot_event(event_df, style=\"full\"):\n    # Create figure and animation\n    fig, ax = plt.subplots()\n    l, = ax.plot([], [])\n    animation = FuncAnimation(fig, func=lambda x: animation_frame(x, event_df, ax, style=style), \n                              frames=event_df[\"frame\"].unique())\n    \n    # Display animation as HTML5 video\n    return HTML(animation.to_html5_video())","metadata":{"execution":{"iopub.status.busy":"2023-03-27T22:39:31.070372Z","iopub.execute_input":"2023-03-27T22:39:31.071250Z","iopub.status.idle":"2023-03-27T22:39:31.112386Z","shell.execute_reply.started":"2023-03-27T22:39:31.071193Z","shell.execute_reply":"2023-03-27T22:39:31.110713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"paths = train.head(5)['path']\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-27T22:47:29.811284Z","iopub.execute_input":"2023-03-27T22:47:29.811746Z","iopub.status.idle":"2023-03-27T22:47:29.825010Z","shell.execute_reply.started":"2023-03-27T22:47:29.811706Z","shell.execute_reply":"2023-03-27T22:47:29.823859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = get_sign_df(paths[0])\nprint(\"SIGN : \"+\"BLOW\")\nplot_event(temp)","metadata":{"execution":{"iopub.status.busy":"2023-03-27T22:48:16.221759Z","iopub.execute_input":"2023-03-27T22:48:16.222310Z","iopub.status.idle":"2023-03-27T22:48:34.560447Z","shell.execute_reply.started":"2023-03-27T22:48:16.222251Z","shell.execute_reply":"2023-03-27T22:48:34.559070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = get_sign_df(paths[1])\nprint(\"SIGN : \"+\"WAIT\")\nplot_event(temp)","metadata":{"execution":{"iopub.status.busy":"2023-03-27T22:48:47.678928Z","iopub.execute_input":"2023-03-27T22:48:47.679415Z","iopub.status.idle":"2023-03-27T22:48:57.043383Z","shell.execute_reply.started":"2023-03-27T22:48:47.679368Z","shell.execute_reply":"2023-03-27T22:48:57.042100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = get_sign_df(paths[2])\nprint(\"SIGN : \"+\"CLOUD\")\nplot_event(temp)","metadata":{"execution":{"iopub.status.busy":"2023-03-27T22:49:16.637332Z","iopub.execute_input":"2023-03-27T22:49:16.638614Z","iopub.status.idle":"2023-03-27T22:50:36.206131Z","shell.execute_reply.started":"2023-03-27T22:49:16.638549Z","shell.execute_reply":"2023-03-27T22:50:36.204497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Adding Sequence Metadata in Train Dataframe**","metadata":{}},{"cell_type":"code","source":"\ndef get_seq_meta(row, invert_y=True, do_counts=False):\n    \"\"\"Calculates and adds metadata to the given row of sign language event data.\n    \n    Args:\n        row (pandas.core.series.Series): A row of sign language event data containing columns:\n            path: The file path to the Parquet file containing the landmark data for the event.\n        invert_y (bool, optional): Whether to invert the y-coordinate of each landmark. Defaults to True.\n    \n    Returns:\n        pandas.core.series.Series: The input row with added metadata columns:\n            start_frame: The frame number of the first frame in the event.\n            end_frame: The frame number of the last frame in the event.\n            total_frames: The number of frames in the event.\n            face_count: The number of landmarks in the 'face' type. [optional]\n            pose_count: The number of landmarks in the 'pose' type. [optional]\n            left_hand_count: The number of landmarks in the 'left_hand' type. [optional]\n            right_hand_count: The number of landmarks in the 'right_hand' type. [optional]\n            x_min: The minimum x-coordinate value of any landmark in the event.\n            x_max: The maximum x-coordinate value of any landmark in the event.\n            y_min: The minimum y-coordinate value of any landmark in the event.\n            y_max: The maximum y-coordinate value of any landmark in the event.\n            z_min: The minimum z-coordinate value of any landmark in the event.\n            z_max: The maximum z-coordinate value of any landmark in the event.\n    \"\"\"\n    # Extract the sign language event data from the Parquet file at the given path\n    df = get_sign_df(row['path'], invert_y=invert_y)\n    \n    # Count the number of landmarks in each type\n    type_counts = df['type'].value_counts(dropna=False).to_dict()\n    nan_counts  = df.groupby(\"type\")[\"x\"].apply(lambda x: x.isna().sum())\n    \n    # Calculate metadata for the event and add it to the input row\n    row['start_frame'] = df['frame'].min()\n    row['end_frame'] = df['frame'].max()\n    row['total_frames'] = df['frame'].nunique()\n    \n    if do_counts:\n        for _type in [\"face\", \"pose\", \"left_hand\", \"right_hand\"]:\n            row[f'{_type}_count'] = type_counts[_type]\n            row[f'{_type}_nan_count'] = nan_counts[_type]\n        \n    for coord in ['x', 'y', 'z']:\n        row[f'{coord}_min'] = df[coord].min()\n        row[f'{coord}_max'] = df[coord].max()\n    \n    return row\n\ntype_kp_map = dict(face=468, left_hand=21, pose=33, right_hand=21)\ncol_order = [\n    'path', 'participant_id', 'sequence_id', 'sign', 'start_frame', 'end_frame', 'total_frames', \n    'face_nan_count', 'face_nan_pct', 'left_hand_nan_count', 'left_hand_nan_pct', 'pose_nan_count', 'pose_nan_pct',\n    'right_hand_nan_count', 'right_hand_nan_pct', 'x_min', 'x_max', 'y_min', 'y_max', 'z_min', 'z_max',\n]","metadata":{"execution":{"iopub.status.busy":"2023-03-27T21:02:50.681805Z","iopub.execute_input":"2023-03-27T21:02:50.682236Z","iopub.status.idle":"2023-03-27T21:02:50.696860Z","shell.execute_reply.started":"2023-03-27T21:02:50.682199Z","shell.execute_reply":"2023-03-27T21:02:50.695563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"PCT_TO_EXAMINE = 1 #100%\nsubsample_train_df = train.sample(frac=PCT_TO_EXAMINE, random_state=20).reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-27T21:27:57.652635Z","iopub.execute_input":"2023-03-27T21:27:57.653799Z","iopub.status.idle":"2023-03-27T21:27:57.680183Z","shell.execute_reply.started":"2023-03-27T21:27:57.653748Z","shell.execute_reply":"2023-03-27T21:27:57.678955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subsample_train_df = subsample_train_df.progress_apply(lambda x: get_seq_meta(x, do_counts=True), axis=1)\nfor _type, _count in type_kp_map.items():\n    subsample_train_df[f\"{_type}_appears_pct\"] = subsample_train_df[f\"{_type}_count\"]/(subsample_train_df[f\"total_frames\"]*_count)\n    subsample_train_df[f\"{_type}_nan_pct\"]     = subsample_train_df[f\"{_type}_nan_count\"]/(subsample_train_df[f\"total_frames\"]*_count)\n# Extended save for later...\nsubsample_train_df.to_csv(\"extended_train.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-27T21:28:29.134530Z","iopub.execute_input":"2023-03-27T21:28:29.135627Z","iopub.status.idle":"2023-03-27T22:37:34.111255Z","shell.execute_reply.started":"2023-03-27T21:28:29.135581Z","shell.execute_reply":"2023-03-27T22:37:34.109674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"extended = pd.read_csv(\"/kaggle/working/extended_train.csv\")\nextended.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-27T22:38:40.518216Z","iopub.execute_input":"2023-03-27T22:38:40.518756Z","iopub.status.idle":"2023-03-27T22:38:41.101487Z","shell.execute_reply.started":"2023-03-27T22:38:40.518707Z","shell.execute_reply":"2023-03-27T22:38:41.100098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Key Takeaways:\n* Right Hand --> 21 Keypoints\n* Left Hand --> 21 Keypoints\n* Pose --> 33 Keypoints\n* Face --> 468 Keypoints\n\n* Sequences can start almost anywhere from frame 0 to frame 484 but the mean is ~30\n* Sequences can end almost anywhere from frame 1 to frame 499 but the mean is ~67\n* Sequences can be different lengths (and are inclusive of their bounds) from a length of 2 to a length of 500. Sequences have a mean length of ~37.5","metadata":{}},{"cell_type":"code","source":"def title_map_fn(ann):\n    title_map = {\n    'face_nan_pct': '<b>Percentage Of <i>Face</i> Data Points That Are NaN</b>', \n    'left_hand_nan_pct': '<b>Percentage Of <i>Left Hand</i> Data Points That Are NaN</b>',\n    'pose_nan_pct': '<b>Percentage Of <i>Pose</i> Data Points That Are NaN</b>',\n    'right_hand_nan_pct': '<b>Percentage Of <i>Right Hand</i> Data Points That Are NaN</b>'}\n    ann.text = title_map.get(ann.text[1:])\n    \nfig = px.histogram(extended, [\"face_nan_pct\", \"left_hand_nan_pct\", \"pose_nan_pct\", \"right_hand_nan_pct\"], height=750,\n                   labels={'variable': '', 'count': '<b>Frequency (LOG)</b>', 'value':\"<b>Percentage of Points That Are NaN</b>\"}, log_y=True, facet_col='variable', nbins=20, opacity=0.75,\n                   facet_col_wrap=2, facet_col_spacing=0.05)\nfig.update_yaxes(title_text='<b>Frequency (LOG)</b>', col=1)\nfig.for_each_annotation(title_map_fn)\nfig.update_layout(showlegend=False)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-27T22:38:45.576387Z","iopub.execute_input":"2023-03-27T22:38:45.576841Z","iopub.status.idle":"2023-03-27T22:38:45.933528Z","shell.execute_reply.started":"2023-03-27T22:38:45.576797Z","shell.execute_reply":"2023-03-27T22:38:45.931877Z"},"trusted":true},"execution_count":null,"outputs":[]}]}