{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n \"Trulli\"\n
\n","metadata":{}},{"cell_type":"markdown","source":"#
Introduction
\n\nObjectives: In creating this notebook, my objectives are:\n \n\n\n\nIsolated Sign Language (ISL): The signs in the dataset represent 250 of the first concepts taught to infants in any language. The goal is to create an isolated sign recognizer to incorporate into educational games for helping hearing parents of Deaf children learn American Sign Language (ASL) [G1]\n
The 5 parameters of ASL are [G2]:\n\n\n\nLandmarks Files: The landmarks were extracted from raw videos with the MediaPipe holistic model [G3]. Not all of the frames necessarily had visible hands or hands that could be detected by the model [G4]. The spatial coordinates of the landmark are normalized to 0 and 1. Any points that are outside of [0, 1] are Mediapipe artifacts [G5].\n ","metadata":{}},{"cell_type":"code","source":"# Install\n!pip install -q itables 2> /dev/null\n!pip install -q flatbuffers 2> /dev/null\n!pip install -q mediapipe 2> /dev/null","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-05T11:00:26.402662Z","iopub.execute_input":"2023-04-05T11:00:26.403558Z","iopub.status.idle":"2023-04-05T11:01:07.229573Z","shell.execute_reply.started":"2023-04-05T11:00:26.403516Z","shell.execute_reply":"2023-04-05T11:01:07.22702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Imports\n\n# activate interactive mode of pd.dataframe\nimport pandas as pd\nfrom itables import init_notebook_mode\ninit_notebook_mode(all_interactive=True, connected=True)\n\nimport os\n\nimport json\nfrom tqdm import tqdm\nimport numpy as np\nimport itertools\n\nimport tensorflow as tf\n\n#pytorch model\nimport torch\nimport torch.nn.functional as F\nimport torch.nn as nn\n\nimport seaborn as sns\nimport mediapipe as mp\nimport matplotlib.pyplot as plt\n\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\n\nfrom matplotlib import animation\nfrom pathlib import Path\nimport IPython\nfrom IPython import display\nfrom IPython.core.display import display, HTML, Javascript\nfrom IPython.display import Markdown as md\n\nimport mediapipe as mp\nfrom mediapipe.framework.formats import landmark_pb2","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-05T11:01:07.232715Z","iopub.execute_input":"2023-04-05T11:01:07.233148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Config\n\nclass Cfg:\n INPUT_ROOT = Path('/kaggle/input/asl-signs/')\n OUTPUT_ROOT = Path('kaggle/working')\n INDEX_MAP_FILE = INPUT_ROOT / 'sign_to_prediction_index_map.json'\n TRAN_FILE = INPUT_ROOT / 'train.csv'\n INDEX = 'sequence_id'\n ROW_ID = 'row_id'\n \nLANDMARK_FILES_DIR = \"/kaggle/input/asl-signs/train_landmark_files\"\nlabel_map = json.load(open(\"/kaggle/input/asl-signs/sign_to_prediction_index_map.json\", \"r\"))\n\ntrain_df = pd.read_csv(\"/kaggle/input/gislr-extended-train-dataframe/extended_train.csv\")\ntrain_df['label'] = train_df['sign'].map(label_map)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Helpers\n\nROWS_PER_FRAME = 543\ndef load_relevant_data_subset(pq_path):\n data_columns = ['x', 'y', 'z']\n data = pd.read_parquet(pq_path, columns=data_columns)\n n_frames = int(len(data) / ROWS_PER_FRAME)\n data = data.values.reshape(n_frames, ROWS_PER_FRAME, len(data_columns))\n return data.astype(np.float32)\n\n# https://www.kaggle.com/code/ted0071/gislr-visualization\ndef read_index_map(file_path=Cfg.INDEX_MAP_FILE):\n \"\"\"Reads the sign to predict as json file.\"\"\"\n with open(file_path, \"r\") as f:\n result = json.load(f)\n return result \n\ndef read_train(file_path=Cfg.TRAN_FILE):\n \"\"\"Reads the train csv as pandas data frame.\"\"\"\n return pd.read_csv(file_path).set_index(Cfg.INDEX)\n\ndef read_landmark_data_by_path(file_path, input_root=Cfg.INPUT_ROOT):\n \"\"\"Reads landmak data by the given file path.\"\"\"\n data = pd.read_parquet(input_root / file_path)\n return data.set_index(Cfg.ROW_ID)\n\ndef read_landmark_data_by_id(sequence_id, train_data):\n \"\"\"Reads the landmark data by the given sequence id.\"\"\"\n file_path = train_data.loc[sequence_id]['path']\n return read_landmark_data_by_path(file_path)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Metadata Analysis
\n\nIn this section, I want to explore the number of sequences, frames, and average frames per sign, and participant. Thus, I split the train data into 2 levels for further analysis: \n\n","metadata":{}},{"cell_type":"markdown","source":"ISL Level\n","metadata":{}},{"cell_type":"code","source":"meta_data_df = train_df.groupby('sign').agg({'sequence_id': 'count',\n 'total_frames': 'sum'})\nmeta_data_df.columns = ['num_seq','num_frames']\nmeta_data_df['avg_frames'] = np.round(meta_data_df['num_frames']/meta_data_df['num_seq'])\n\nmeta_data_df","metadata":{"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"int(meta_data_df.num_seq.sum()/250), int(meta_data_df.avg_frames.sum()/250)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure()\n\ncols_name = meta_data_df.columns.values.tolist()\ncolors = [\"#0F9D58\",\"#4285F4\",\"#F4B400\"]\n\nfor color,col in zip(colors,cols_name):\n # store tmp df\n tmp = meta_data_df.sort_values(col)\n fig.add_trace(go.Bar(x=tmp.index, \n y=tmp[col],\n width=0.5,name=col,marker_color=color))\n\nfig.update_layout(\n title={\n 'text': \"ISL Distribution: All Traces\",\n 'font': dict(size=20,family=\"Georgia\",color=colors[1]),\n 'y':0.87,\n 'x':0.035,\n 'xanchor': 'left',\n 'yanchor': 'top'},\n template=\"plotly_white\",\n xaxis_tickangle=-45,\n width= 4000,\n xaxis=dict(title='Sign', fixedrange=True),\n yaxis=dict(title='Count',fixedrange=True),\n showlegend=True,\n \n updatemenus=[\n dict(\n # customize dropdown\n active=0,\n direction=\"down\",\n pad={\"r\": 50, \"t\": 25},\n showactive=True,\n x=0.005,\n xanchor=\"right\",\n y=1.2,\n yanchor=\"top\",\n \n # customize button\n buttons=list([\n dict(label=\"All\",\n method=\"update\",\n args=[{\"visible\": [True, True,True]},\n {\"title\": \"ISL Distribution: All Traces\",\n \"legend\":True,\n }]),\n dict(label=\"Sequences\",\n method=\"update\",\n args=[{\"visible\": [True, False,False]},\n {\"title\": \"Distribution of Number of Sequences per Sign\",\n \"legend\":True,\n }]),\n dict(label=\"Frames\",\n method=\"update\",\n args=[{\"visible\": [False,True, False]},\n {\"title\": \"Distribution of Number of Frames per Sign\",\n \"legend\":True,\n }]),\n dict(label=\"Avg frames\",\n method=\"update\",\n args=[{\"visible\": [False, False,True]},\n {\"title\": \"Distribution of Average Frames per Sequence per Sign\",\n \"legend\":True,\n }]),\n ]),\n ),\n ])\n\nfig.show(config= dict(displayModeBar = False))","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\nObservations: \n \n\n\n
","metadata":{}},{"cell_type":"markdown","source":"Participant Level","metadata":{}},{"cell_type":"code","source":"participant_level_df = train_df.groupby(['participant_id','sign']).agg({'sequence_id': 'count',\n 'total_frames': 'sum'})\nparticipant_level_df['avg_frames'] = np.round(participant_level_df.total_frames/participant_level_df.sequence_id)\nparticipant_level_df.columns = ['num_seq','num_frames','avg_frames']\nparticipant_level_df = participant_level_df.reset_index()\nparticipant_level_df","metadata":{"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"int(participant_level_df.num_seq.sum()/21), int(participant_level_df.num_frames.sum()/21)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure()\n\ncols_name = participant_level_df.columns.values.tolist()[2:]\ncolors = [\"#0F9D58\",\"#4285F4\",\"#F4B400\"]\nmethods = ['sum','sum','sum']\n\nfor color,col,method in zip(colors,cols_name,methods):\n # store tmp series object\n tmp = participant_level_df.groupby(['participant_id'])\\\n .agg({col: method})[col]\\\n .sort_values(ascending=True)\n fig.add_trace(go.Bar(x=tmp.index.astype('str'), \n y=tmp.values,\n width=0.8,name=col,marker_color=color))\nfig.update_layout(\n title={\n 'text': \"Participant Distribution: All Traces\",\n 'font': dict(size=20,family=\"Georgia\",color=colors[1]),\n 'y':0.87,\n 'x':0.18,\n 'xanchor': 'left',\n 'yanchor': 'top'},\n template=\"plotly_white\",\n xaxis_tickangle=-45,\n width= 800,\n height=500,\n xaxis=dict(title='Participant_ID', fixedrange=True),\n yaxis=dict(title='Count',fixedrange=True),\n showlegend=True,\n \n updatemenus=[\n dict(\n # customize dropdown\n active=0,\n direction=\"down\",\n pad={\"r\": 50, \"t\": 25},\n showactive=True,\n x=0.01,\n xanchor=\"right\",\n y=1.20,\n yanchor=\"top\",\n \n # customize button\n buttons=list([\n dict(label=\"All\",\n method=\"update\",\n args=[{\"visible\": [True, True,True]},\n {\"title\": \"Participant Distribution: All Traces\",\n \"legend\":True,\n }]),\n dict(label=\"Sequences\",\n method=\"update\",\n args=[{\"visible\": [True, False,False]},\n {\"title\": \"Distribution of Number of Sequences per Participant\",\n \"legend\":True,\n }]),\n dict(label=\"Frames\",\n method=\"update\",\n args=[{\"visible\": [False,True, False]},\n {\"title\": \"Distribution of Number of Frames per Participant\",\n \"legend\":True,\n }]),\n dict(label=\"Avg frames\",\n method=\"update\",\n args=[{\"visible\": [False, False,True]},\n {\"title\": \"Distribution of Average Frames per Participant\",\n \"legend\":True,\n }]),\n ]),\n ),\n ])\n\nfig.show(config= dict(displayModeBar = False))","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\nObservations: \n \n\n\n
","metadata":{}},{"cell_type":"markdown","source":"#
Landmarks Analysis
\n\n\nIn this section, I want to visualize and gain more insights from individual frames for each sequence. Thus, I adjust some helper functions from Roland Abel [C1] and create another helper function for interactive landmarks visualization.
Below are some references for hand, full body, and face landmarks: \n\nHand Landmarks\n\n![Hand Landmarks](https://developers.google.com/static/mediapipe/images/solutions/hand-landmarks.png)","metadata":{}},{"cell_type":"markdown","source":"Full Body Landmarks\n![Full Body Landmarks](https://mediapipe.dev/images/mobile/pose_tracking_full_body_landmarks.png)\n\n\nFor ASL, the upper body landmarks are more important than the lower body landmarks.","metadata":{}},{"cell_type":"markdown","source":"Face Landmarks (Contours)\n\n\nFor ASL, the non-manual signals such as head nod, eyebrows, nose, eyes, and lips are used to convey additional meaning with a sign. Thus, instead of visualizing the facemesh tesselation, I decide to only use facemesh contours.
Face Landmarks to use: \n\n\n","metadata":{}},{"cell_type":"code","source":"# Helper Functions\n# [C1] adjusted from Roland Abel: https://www.kaggle.com/code/ted0071/gislr-visualization\ntrain_data = read_train()\n\nmp_drawing = mp.solutions.drawing_utils\nmp_hands = mp.solutions.hands\nmp_face_mesh = mp.solutions.face_mesh\nmp_pose = mp.solutions.pose\n\n# contour connections\nCONTOURS = list(itertools.chain(*mp_face_mesh.FACEMESH_CONTOURS))\n\ndef create_blank_image(height, width):\n return np.zeros((height, width, 3), np.uint8)\n\ndef draw_landmarks(data, image, frame_id, \n landmark_type, connection_type, \n landmark_color=(255, 0, 0), connection_color=(0, 20, 255), \n thickness=2, circle_radius=1):\n \"\"\"Draws landmarks\"\"\"\n df = data.groupby(['frame', 'type']).get_group((frame_id, landmark_type)).copy()\n if landmark_type == 'face':\n df.loc[~df['landmark_index'].isin(CONTOURS),'x'] = float('NaN') #-1*df[~df['landmark_index'].isin(CONTOURS)]['x'].values\n\n \n landmarks = [landmark_pb2.NormalizedLandmark(x=lm.x, y=lm.y, z=lm.z) for idx, lm in df.iterrows()]\n landmark_list = landmark_pb2.NormalizedLandmarkList(landmark = landmarks)\n #print(len(landmark_list.landmark))\n mp_drawing.draw_landmarks(\n image=image,\n landmark_list=landmark_list, \n connections=connection_type,\n landmark_drawing_spec=mp_drawing.DrawingSpec(\n color=landmark_color, \n thickness=thickness, \n circle_radius=circle_radius),\n connection_drawing_spec=mp_drawing.DrawingSpec(\n color=connection_color, \n thickness=thickness, \n circle_radius=circle_radius))\n return image","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_ids(df, row):\n participant_id = df.participant_id.values[row]\n sequence_id = df.sequence_id.values[row]\n \n return participant_id, sequence_id\n\ndef draw_data(participant_id, sequence_id, train_data):\n height = 700\n width = 500\n\n # Read and get frames\n data = read_landmark_data_by_id(sequence_id, train_data)\n frame_ids = data.frame.unique().tolist()\n buttons_ids = []\n buttons_seq_ids = []\n buttons=[]\n\n fig = make_subplots(rows=2, cols=3,\n specs=[[{}, {},{\"rowspan\": 2}],\n [{}, {},None]],\n vertical_spacing=0.1,\n subplot_titles=('Face', 'Pose',\n 'All', 'Left Hand',\n 'Right Hand'),\n print_grid=False)\n\n buttons_seq_ids.append(dict(label=f\"{sequence_id}\",\n method=\"restyle\",\n args=[{\"visible\": None}]\n ))\n buttons_ids.append(dict(label=f\"{participant_id}\",\n method=\"restyle\",\n args=[{\"visible\": None}]\n ))\n\n for i,frame_id in enumerate(frame_ids): \n r_hand = draw_landmarks(data, image=create_blank_image(height, width ), \n frame_id=frame_id,\n landmark_type = 'right_hand', \n connection_type = mp_hands.HAND_CONNECTIONS,\n landmark_color=(255, 0, 0),\n connection_color=(0, 20, 255), \n thickness=3, \n circle_radius=3)\n\n\n l_hand = draw_landmarks(data, image=create_blank_image(height, width), \n frame_id=frame_id,\n landmark_type = 'left_hand', \n connection_type = mp_hands.HAND_CONNECTIONS,\n landmark_color=(255, 0, 0),\n connection_color=(0, 20, 255), \n thickness=3, \n circle_radius=3)\n\n\n\n face = draw_landmarks(data, image=create_blank_image(height, width), \n frame_id=frame_id,\n landmark_type='face', \n connection_type= mp_face_mesh.FACEMESH_CONTOURS,\n landmark_color=(255, 255, 255),\n connection_color=(0, 255, 0),\n thickness=1, \n circle_radius=1)\n\n pose = draw_landmarks(data, image=create_blank_image(height, width), \n frame_id=frame_id,\n landmark_type='pose', \n connection_type= mp_pose.POSE_CONNECTIONS,\n landmark_color=(255, 255, 255),\n connection_color=(255, 0, 0),\n thickness=2, \n circle_radius=2)\n\n fig.add_trace(px.imshow(face).data[0], row=1, col=1)\n fig.add_trace(px.imshow(pose).data[0], row=1, col=2)\n fig.add_trace(px.imshow(l_hand).data[0], row=2, col=1)\n fig.add_trace(px.imshow(r_hand).data[0], row=2, col=2)\n fig.add_trace(px.imshow(face+pose+l_hand+r_hand, aspect='auto').data[0], row=1, col=3)\n\n visible=[False,False,False,False,False]*len(frame_ids)\n visible[i*5:i*5+5]=[True]*5\n buttons.append(dict(label=f\"{frame_id}\",\n method=\"update\",\n args=[{\"visible\": visible}])) \n\n sign = train_df.query('sequence_id == @sequence_id')['sign'].values[0]\n\n fig.update_layout(\n title={\n 'text': f'Sign: {sign}',\n 'font': dict(size=20,family=\"Georgia\",color=colors[1]),\n 'y':0.98,\n 'x':0.5,\n 'xanchor': 'center',\n 'yanchor': 'top'},\n\n\n template=\"plotly_white\",\n width= 800,\n height=600,\n showlegend=True,\n\n\n updatemenus=[\n # Participant_ID\n dict(\n # customize dropdown\n active=0,\n direction=\"down\",\n pad={\"r\": 50, \"t\": 25},\n showactive=True,\n x=0.1,\n xanchor=\"left\",\n y=1.2,\n yanchor=\"top\",\n\n # customize button \n buttons=buttons_ids),\n\n # Sequence_ID\n dict(\n # customize dropdown\n active=0,\n direction=\"down\",\n pad={\"r\": 50, \"t\": 25},\n showactive=True,\n x=0.43,\n xanchor=\"left\",\n y=1.2,\n yanchor=\"top\",\n\n # customize button \n buttons=buttons_seq_ids),\n\n # Frames_ID\n dict(\n # customize dropdown\n active=0,\n direction=\"down\",\n pad={\"r\": 50, \"t\": 25},\n showactive=True,\n x=0.8,\n xanchor=\"left\",\n y=1.2,\n yanchor=\"top\",\n\n # customize button \n buttons=buttons),\n\n ])\n\n fig.update_xaxes(showticklabels=False,fixedrange=True)\n fig.update_yaxes(showticklabels=False,fixedrange=True)\n\n fig.add_annotation(text=\"Participant_ID\", x=-0.05, xref=\"paper\", y=1.12, yref=\"paper\",\n align=\"left\", showarrow=False)\n fig.add_annotation(text=\"Sequence_ID\", x=0.35, xref=\"paper\", y=1.125, yref=\"paper\",\n align=\"left\", showarrow=False)\n fig.add_annotation(text=\"Frame_ID\", x=0.78, xref=\"paper\", y=1.13, yref=\"paper\",\n align=\"left\", showarrow=False)\n \n return fig","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Landmarks Visualization","metadata":{}},{"cell_type":"code","source":"sign_table = {}\nfor i in range(25):\n sign_table[f'{i}'] = train_df.sign.unique()[i*10:i*10+10].tolist()\nsign_table = pd.DataFrame(sign_table)\nsign_table","metadata":{"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Example 1: Wait","metadata":{}},{"cell_type":"markdown","source":"> Disclaimer: I want to produce an interactive visualization based on Participant_ID, Sequence_ID, and Frame_ID. However, I haven't figured out the way to do the chained dropdown callback without using Dash Plottly yet. Thus, only the Frame_ID dropdown is active, the Participant_ID and Sequence_ID dropdowns are currently inactive ^^'. ","metadata":{}},{"cell_type":"code","source":"participant_id, sequence_id = get_ids(train_df[train_df['sign'] == 'wait'], 10)\nfig = draw_data(participant_id,sequence_id,train_data)\nfig.show(config= dict(displayModeBar = False))","metadata":{"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Example 2: Cloud","metadata":{}},{"cell_type":"code","source":"participant_id, sequence_id = get_ids(train_df[train_df['sign'] == 'cloud'], 10 )\nfig = draw_data(participant_id,sequence_id,train_data)\nfig.show(config= dict(displayModeBar = False))","metadata":{"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Example 3: Flower","metadata":{}},{"cell_type":"code","source":"participant_id, sequence_id = get_ids(train_df[train_df['sign'] == 'flower'], 10 )\nfig = draw_data(participant_id,sequence_id,train_data)\nfig.show(config= dict(displayModeBar = False))","metadata":{"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\nObservations: \n\n\n\n=> Besides Mediapipe artifacts (normalized coordinates out of range [0,1]), we need to deal with NaN coordinates as well!\n
","metadata":{}},{"cell_type":"markdown","source":"#
Feature Processing
\n\nProcessing: \n\nIn this section, I want to visualize and implement a feature preprocessing, inspired by Robert Hatch [C2], Darien Schettler [C3] , and Heng CK [C4] public notebooks.
There are 3 main steps in my approach (Fig 1): \n\nStep 1: Data Normalization\n\nStep 2: Landmarks Reduction\n\nStep 3: Frames Interpolation\n\n![](https://drive.google.com/uc?id=1UxXL7aTg_YpSo8-nPV019hk6rLmXZRiz)\n\n","metadata":{}},{"cell_type":"code","source":"# Clean representation for frame idx map inspired by Darien Schettler [C3]\nIDX_MAP = {\"contours\" : list(set(CONTOURS)),\n \"left_hand\" : np.arange(468, 489).tolist(),\n \"upper_body\" : np.arange(489, 511).tolist(),\n \"right_hand\" : np.arange(522, 543).tolist()}\n\nFIXED_FRAMES = 37 # based on the above observations","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class FeaturePreprocess(nn.Module):\n def __init__(self):\n super().__init__()\n \n def forward(self, x_in):\n n_frames = x_in.shape[0]\n\n # Normalization to a common mean by Heng CK [C4]\n x_in = x_in - x_in[~torch.isnan(x_in)].mean(0,keepdim=True) \n x_in = x_in / x_in[~torch.isnan(x_in)].std(0, keepdim=True)\n\n # Landmarks reduction\n contours = x_in[:, IDX_MAP['contours']]\n lhand = x_in[:, IDX_MAP['left_hand']]\n pose = x_in[:, IDX_MAP['upper_body']]\n rhand = x_in[:, IDX_MAP['right_hand']]\n \n x_in = torch.cat([contours,\n lhand,\n pose,\n rhand], 1) # (n_frames, 192, 3)\n \n # Replace nan with 0 before Interpolation\n x_in[torch.isnan(x_in)] = 0\n \n # Frames interpolation inspired by Robert Hatch [C2]\n # If n_frames < k, use linear interpolation,\n # else, use nearest neighbor interpolation\n x_in = x_in.permute(2,1,0) #(3, 192, n_frames)\n if n_frames < FIXED_FRAMES:\n x_in = F.interpolate(x_in, size=(FIXED_FRAMES), mode= 'linear')\n else:\n x_in = F.interpolate(x_in, size=(FIXED_FRAMES), mode= 'nearest-exact')\n \n return x_in.permute(2,1,0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_in = torch.tensor(load_relevant_data_subset(train_df.path[0]))\nfeature_preprocess = FeaturePreprocess()\nfeature_preprocess(x_in).shape, x_in[0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Save Preprocessed Features","metadata":{}},{"cell_type":"code","source":"# adapted and adjusted from Robert Hatch [C2]\nright_handed_signer = [26734, 28656, 25571, 62590, 29302, \n 49445, 53618, 18796, 4718, 2044, \n 37779, 30680]\nleft_handed_signer = [16069, 32319, 36257, 22343, 27610, \n 61333, 34503, 55372, ]\nboth_hands_signer = [37055 ]\nmessy = [29302 ]\n\ndef convert_row(row, right_handed=True):\n x = torch.tensor(load_relevant_data_subset(row[1].path))\n x = feature_preprocess(x).cpu().numpy()\n return x, row[1].label\n\ndef convert_and_save_data(df):\n total = df.shape[0]\n npdata = np.zeros((total, 37, 192 ,3))\n nplabels = np.zeros(total)\n for i, row in tqdm(enumerate(df.iterrows()), total=total):\n (x,y) = convert_row(row)\n npdata[i,:,:,:] = x\n nplabels[i] = y\n \n np.save(\"feature_data.npy\", npdata)\n np.save(\"feature_labels.npy\", nplabels)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"convert_and_save_data(train_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = np.load(\"feature_data.npy\")\ny = np.load(\"feature_labels.npy\")\nprint(X.shape, y.shape)\nprint(X[0].shape,y[0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n


If you find this notebook useful, do give me an upvote, it motivates me a lot.
This notebook is still a work in progress. Keep checking for further developments!😊
Thank you!

","metadata":{}},{"cell_type":"markdown","source":"#
Acknowledgement
\n\nI want to thank Darien Schettler, Robert Hatch, Roland Abel, Heng CK, and many other Kagglers for their contribution to this competition. Their work has helped me better understand the data and how to take it to the next step.","metadata":{}},{"cell_type":"markdown","source":"#
References
\n\n**General References (G):**\n\n[G1] [GISLR: Data Card](https://www.kaggle.com/competitions/asl-signs/overview/data-card)\n\n[G2] [The 5 Parameters of ASL](https://www.mtsac.edu/llc/passportrewards/languagepartners/5ParametersofASL.pdf)\n\n[G3] [MediaPipe holistic model](https://google.github.io/mediapipe/solutions/holistic.html)\n\n[G4] [GISLR: Dataset Description](https://www.kaggle.com/competitions/asl-signs/data)\n\n[G5] [Discussion: X-Y coordinates aren't normalized to 0-1](https://www.kaggle.com/competitions/asl-signs/discussion/392286)\n\n[G6] [Mediapipe: Tensors to Face Landmarks](https://github.com/google/mediapipe/blob/master/mediapipe/modules/face_landmark/tensors_to_face_landmarks_with_attention.pbtxt)\n\n[G7] [Mediapipe: Facemesh Connections](https://github.com/google/mediapipe/blob/master/mediapipe/python/solutions/face_mesh_connections.py)\n\n**Code References (C):**\n\n[C1] [Roland Abel: GISLR | Visualization 🤟](https://www.kaggle.com/code/ted0071/gislr-visualization)\n\n[C2] [Robert Hatch: GISLR Feature Data: On the Shoulders](https://www.kaggle.com/code/roberthatch/gislr-feature-data-on-the-shoulders/notebook)\n\n[C3] [Darien Schettler: 🤟 GISLR 🤟 - 📚Learn – 🔭EDA – 🤖Baseline](https://www.kaggle.com/code/dschettler8845/gislr-learn-eda-baseline)\n\n[C4] [Heng CK: [LB 0.62] pytorch transformer solution](https://www.kaggle.com/code/hengck23/lb-0-62-pytorch-transformer-solution/notebook)\n","metadata":{}}]}