{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!ls ../input/asl-signs/ -GFlash --color","metadata":{"execution":{"iopub.status.busy":"2023-04-04T05:02:35.580688Z","iopub.execute_input":"2023-04-04T05:02:35.581161Z","iopub.status.idle":"2023-04-04T05:02:36.744876Z","shell.execute_reply.started":"2023-04-04T05:02:35.581120Z","shell.execute_reply":"2023-04-04T05:02:36.742559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%capture\n!pip install nb_black # for formatting the code","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-04-04T05:45:40.871072Z","iopub.execute_input":"2023-04-04T05:45:40.871496Z","iopub.status.idle":"2023-04-04T05:45:53.352841Z","shell.execute_reply.started":"2023-04-04T05:45:40.871460Z","shell.execute_reply":"2023-04-04T05:45:53.350861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%load_ext lab_black","metadata":{"execution":{"iopub.status.busy":"2023-04-04T05:51:08.501048Z","iopub.execute_input":"2023-04-04T05:51:08.501517Z","iopub.status.idle":"2023-04-04T05:51:08.518491Z","shell.execute_reply.started":"2023-04-04T05:51:08.501479Z","shell.execute_reply":"2023-04-04T05:51:08.516975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%capture\n!pip install itables # for interactive tables","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-04-04T05:03:14.337249Z","iopub.execute_input":"2023-04-04T05:03:14.337784Z","iopub.status.idle":"2023-04-04T05:03:28.087896Z","shell.execute_reply.started":"2023-04-04T05:03:14.337740Z","shell.execute_reply":"2023-04-04T05:03:28.086261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MAIN_PATH = '/kaggle/input/asl-signs'\nEXTENDED_PATH = '/kaggle/input/gislr-extended-train-dataframe'","metadata":{"execution":{"iopub.status.busy":"2023-04-04T05:03:40.474767Z","iopub.execute_input":"2023-04-04T05:03:40.475261Z","iopub.status.idle":"2023-04-04T05:03:40.489748Z","shell.execute_reply.started":"2023-04-04T05:03:40.475215Z","shell.execute_reply":"2023-04-04T05:03:40.487966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np # linear algebra\n\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom itables import init_notebook_mode\n\ninit_notebook_mode(all_interactive=True, connected=True)\n\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\nimport os\nimport json","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-04-04T05:51:14.400565Z","iopub.execute_input":"2023-04-04T05:51:14.401163Z","iopub.status.idle":"2023-04-04T05:51:14.426260Z","shell.execute_reply.started":"2023-04-04T05:51:14.401113Z","shell.execute_reply":"2023-04-04T05:51:14.424748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Introduction
\n\nObjectives: In creating this notebook, my objectives are:\n \n\n\n- To learn about the data by exploration and visualization
\n\n- To perform some processing techniques for further development
\n\n
\n\n\nIsolated Sign Language (ISL): The signs in the dataset represent 250 of the first concepts taught to infants in any language. The goal is to create an isolated sign recognizer to incorporate into educational games for helping hearing parents of Deaf children learn American Sign Language (ASL) [G1]\n
The 5 parameters of ASL are [G2]:\n\n\n\n- Handshapes
\n\n- Palm Orientations
\n \n- Locations
\n \n- Movements
\n \n- Non-Manual Signals (NMS)
\n\n
\n\nLandmarks Files: The landmarks were extracted from raw videos with the MediaPipe holistic model [G3]. Not all of the frames necessarily had visible hands or hands that could be detected by the model [G4]. The spatial coordinates of the landmark are normalized to 0 and 1. Any points that are outside of [0, 1] are Mediapipe artifacts [G5].\n ","metadata":{}},{"cell_type":"markdown","source":"## Explore train.csv\n- `path` - The path to the landmark file.\n- `participant_id` - A unique identifier for the data contributor.\n- `sequence_id` - A unique identifier for the landmark sequence.\n- `sign` - The label for the landmark sequence.\n\nI also have mapped the sign with it's corresponding label using `map` function.","metadata":{"execution":{"iopub.status.busy":"2023-03-17T10:48:16.721711Z","iopub.execute_input":"2023-03-17T10:48:16.722266Z","iopub.status.idle":"2023-03-17T10:48:16.731646Z","shell.execute_reply.started":"2023-03-17T10:48:16.722218Z","shell.execute_reply":"2023-03-17T10:48:16.729410Z"}}},{"cell_type":"code","source":"train = pd.read_csv(os.path.join(EXTENDED_PATH, \"extended_train.csv\"))\n\nlabel_map = json.load(\n open(\"/kaggle/input/asl-signs/sign_to_prediction_index_map.json\", \"r\")\n)\n\ntrain[\"label\"] = train[\"sign\"].map(label_map)\ntrain","metadata":{"execution":{"iopub.status.busy":"2023-04-04T05:51:21.320973Z","iopub.execute_input":"2023-04-04T05:51:21.321786Z","iopub.status.idle":"2023-04-04T05:51:21.939420Z","shell.execute_reply.started":"2023-04-04T05:51:21.321742Z","shell.execute_reply":"2023-04-04T05:51:21.938032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Sanity check!!
\nCheck for null values","metadata":{}},{"cell_type":"code","source":"train.isnull().sum().any()","metadata":{"execution":{"iopub.status.busy":"2023-04-04T05:04:48.111666Z","iopub.execute_input":"2023-04-04T05:04:48.112684Z","iopub.status.idle":"2023-04-04T05:04:48.148742Z","shell.execute_reply.started":"2023-04-04T05:04:48.112616Z","shell.execute_reply":"2023-04-04T05:04:48.147316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Explore sign_to_prediction_index_map.json\n\nThis contains the index for each sign.","metadata":{}},{"cell_type":"code","source":"with open (file = os.path.join(MAIN_PATH, 'sign_to_prediction_index_map.json'), mode='r') as file:\n data = json.load(file)\n \nindex_df = pd.DataFrame(data.items(), columns=['Object', 'Index'])\n\n# print(data)\nindex_df","metadata":{"execution":{"iopub.status.busy":"2023-04-04T05:49:31.159518Z","iopub.execute_input":"2023-04-04T05:49:31.159998Z","iopub.status.idle":"2023-04-04T05:49:31.199636Z","shell.execute_reply.started":"2023-04-04T05:49:31.159957Z","shell.execute_reply":"2023-04-04T05:49:31.198137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Explore the train landmark files\n\n`train_landmark_files/[participant_id]/[sequence_id].parquet` The landmark data. The landmarks were extracted from raw videos with the MediaPipe holistic model. Not all of the frames necessarily had visible hands or hands that could be detected by the model.\n\nThe data desctiption is as follows :
\n- `frame` - The frame number in the raw video.\n- `row_id` - A unique identifier for the row.\n- `type` - The type of landmark. One of `['face', 'left_hand', 'pose', 'right_hand'].`\n- `landmark_index` - The landmark index number. Details of the hand landmark locations can be found here.\n- `[x/y/z]` - The normalized spatial coordinates of the landmark. These are the only columns that will be provided to your submitted model for inference. The MediaPipe model is not fully trained to predict depth so you may wish to ignore the z values.","metadata":{}},{"cell_type":"markdown","source":"Let's visualize one of the .parquet file","metadata":{}},{"cell_type":"code","source":"parquet_file_0 = pd.read_parquet(os.path.join(EXTENDED_PATH, train['path'][1]))\nparquet_file_0","metadata":{"execution":{"iopub.status.busy":"2023-04-04T05:05:20.793795Z","iopub.execute_input":"2023-04-04T05:05:20.794235Z","iopub.status.idle":"2023-04-04T05:05:21.029559Z","shell.execute_reply.started":"2023-04-04T05:05:20.794199Z","shell.execute_reply":"2023-04-04T05:05:21.028192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"parquet_file_0.shape","metadata":{"execution":{"iopub.status.busy":"2023-04-04T05:05:30.116907Z","iopub.execute_input":"2023-04-04T05:05:30.117480Z","iopub.status.idle":"2023-04-04T05:05:30.132060Z","shell.execute_reply.started":"2023-04-04T05:05:30.117434Z","shell.execute_reply":"2023-04-04T05:05:30.130138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Explore metedata of the files
\n\n\n Firstly I will segregate the dataframe based on sign andthen run the agg function to count the number of sequences, number of frames and the averege frames of the entire dataset.\n","metadata":{}},{"cell_type":"code","source":"meta_data_df = train.groupby('sign').agg({'sequence_id': 'count', 'total_frames': 'sum'})\nmeta_data_df.columns = ['num_seq', 'num_frames']\nmeta_data_df['avg_frames'] = np.round(meta_data_df['num_frames']/meta_data_df['num_seq'])\n\nmeta_data_df","metadata":{"execution":{"iopub.status.busy":"2023-04-04T05:49:36.830626Z","iopub.execute_input":"2023-04-04T05:49:36.831095Z","iopub.status.idle":"2023-04-04T05:49:36.896456Z","shell.execute_reply.started":"2023-04-04T05:49:36.831054Z","shell.execute_reply":"2023-04-04T05:49:36.894639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'The averege number of sqeuences per sign are {round(meta_data_df.num_seq.mean())}.')\nprint(f'The averege frames per sequence are {round(meta_data_df.avg_frames.mean())}.')","metadata":{"execution":{"iopub.status.busy":"2023-04-04T05:08:22.786348Z","iopub.execute_input":"2023-04-04T05:08:22.786795Z","iopub.status.idle":"2023-04-04T05:08:22.805516Z","shell.execute_reply.started":"2023-04-04T05:08:22.786757Z","shell.execute_reply":"2023-04-04T05:08:22.803758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Plot the metadata of the file
","metadata":{"_kg_hide-input":true}},{"cell_type":"code","source":"fig = go.Figure()\n\ncol_name = meta_data_df.columns.values.tolist()\ncolors = [\"#0F9D58\", \"#4285F4\", \"#F4B400\"]\n\nfor color, col in zip(colors, col_name):\n tmp = meta_data_df.sort_values(col)\n fig.add_trace(\n go.Bar(x=tmp.index, y=tmp[col], marker_color=color, name=col, width=0.5)\n )\n\n\nfig.update_layout(\n title={\n \"text\": \"Sign Language Distribution \",\n \"font\": dict(size=20, family=\"Georgia\", color=colors[1]),\n \"y\": 0.87,\n \"x\": 0.035,\n \"xanchor\": \"center\",\n \"yanchor\": \"top\",\n },\n template=\"plotly_white\",\n xaxis_tickangle=-45,\n width=4000,\n xaxis=dict(title=\"Sign\", fixedrange=True),\n yaxis=dict(title=\"Count\", fixedrange=True),\n showlegend=True,\n)\nfig.show(config=dict(displayModeBar=False))","metadata":{"execution":{"iopub.status.busy":"2023-04-04T05:51:41.055328Z","iopub.execute_input":"2023-04-04T05:51:41.056575Z","iopub.status.idle":"2023-04-04T05:51:41.177931Z","shell.execute_reply.started":"2023-04-04T05:51:41.056509Z","shell.execute_reply":"2023-04-04T05:51:41.176750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n We can get some important insights from the above visualization\n
\n - Number of sequences are fairly equally distributed
\n - The sign Mitten has the largest number of frames and average frames pre sequence
\n
\n
\n We can get some important insights from the above visualization:\n
\n - Participant with ID 49445 has largest number of frames & participant 37779 has least number of frames
\n - Three participants have least number of sequences.
\n - On an average there are 170666 frames per participant and there are 4498 sequences per participant.
\n
\n