{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Interactive 3D Animated Visualization of ASL using Plotly! ","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"The aim of this notebook is to provide a powerful tool for visualizing and analyzing 3D ASL using the data provided in the competition. The notebook consists of several helper functions and one main function that is responsible to generate a 3D visualization using Plotly. **The function only takes as input any of the parquet data (converted to a dataframe)** containing the x, y, z coordinates of each landmark.\n\nThis will allow for a more intuitive understanding of the data provided for augmentation, as it shows the hands, pose and face landmarks from different angles and allows for rotation and zooming. This is particularly important for ASL recognition, as hand shape and orientation are key components of sign language.\n\nAdditionally, the 3D visualization can be used for data exploration and analysis. The ability to rotate and zoom in on the hand can reveal patterns and relationships in the data that may not be apparent in a 2D representation. This can help you kagglers better understand the characteristics of different poses and improve our ASL recognition models.\n\nSneak peak 👀👇","metadata":{}},{"cell_type":"markdown","source":"<img src=\"https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExY2FkMzA2OWYxZTA3MzQ4OGQ1ODIyYWRlZmE4ZDBhYjVlMTZkNmQ0OCZjdD1n/IDMyhLF7HMqEEk7eIB/giphy.gif\">","metadata":{}},{"cell_type":"code","source":"# import the desired packages\nimport numpy as np\nimport pandas as pd\nimport json\nimport plotly.graph_objects as go\nimport plotly.io as pio\npio.templates.default = \"simple_white\"","metadata":{"execution":{"iopub.status.busy":"2023-03-07T14:07:22.485450Z","iopub.execute_input":"2023-03-07T14:07:22.486055Z","iopub.status.idle":"2023-03-07T14:07:24.194326Z","shell.execute_reply.started":"2023-03-07T14:07:22.485998Z","shell.execute_reply":"2023-03-07T14:07:24.192773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helper Functions ","metadata":{}},{"cell_type":"markdown","source":"These two functions are called within the main function to assign specific colors to different landmark types as well and for correctly connecting landmarks between one another. ","metadata":{}},{"cell_type":"code","source":"# assign desired colors to landmarks\ndef assign_color(row):\n    if row == 'face':\n        return 'red'\n    elif 'hand' in row:\n        return 'dodgerblue'\n    else:\n        return 'green'\n\n# specifies the plotting order\ndef assign_order(row):\n    if row.type == 'face':\n        return row.landmark_index + 101\n    elif row.type == 'pose':\n        return row.landmark_index + 30\n    elif row.type == 'left_hand':\n        return row.landmark_index + 80\n    else:\n        return row.landmark_index","metadata":{"execution":{"iopub.status.busy":"2023-03-07T14:07:42.009695Z","iopub.execute_input":"2023-03-07T14:07:42.010424Z","iopub.status.idle":"2023-03-07T14:07:42.016743Z","shell.execute_reply.started":"2023-03-07T14:07:42.010380Z","shell.execute_reply":"2023-03-07T14:07:42.015243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Main Function ","metadata":{}},{"cell_type":"code","source":"def visualise3d_landmarks(parquet_df):\n    connections = [  # right hand\n        [0, 1, 2, 3, 4, ],\n        [0, 5, 6, 7, 8],\n        [0, 9, 10, 11, 12],\n        [0, 13, 14, 15, 16],\n        [0, 17, 18, 19, 20],\n\n        # pose\n        [38, 36, 35, 34, 30, 31, 32, 33, 37],\n        [40, 39],\n        [52, 46, 50, 48, 46, 44, 42, 41, 43, 45, 47, 49, 45, 51],\n        [42, 54, 56, 58, 60, 62, 58],\n        [41, 53, 55, 57, 59, 61, 57],\n        [54, 53],\n\n        # left hand\n        [80, 81, 82, 83, 84, ],\n        [80, 85, 86, 87, 88],\n        [80, 89, 90, 91, 92],\n        [80, 93, 94, 95, 96],\n        [80, 97, 98, 99, 100], ]\n\n    frames = sorted(set(parquet_df.frame))\n    first_frame = min(frames)\n    parquet_df['color'] = parquet_df.type.apply(lambda row: assign_color(row))\n    parquet_df['plot_order'] = parquet_df.apply(lambda row: assign_order(row), axis=1)\n    first_frame_df = parquet_df[parquet_df.frame == first_frame].copy()\n    first_frame_df = first_frame_df.sort_values([\"plot_order\"]).set_index('plot_order')\n\n\n    frames_l = []\n    for frame in frames:\n        filtered_df = parquet_df[parquet_df.frame == frame].copy()\n        filtered_df = filtered_df.sort_values([\"plot_order\"]).set_index(\"plot_order\")\n        traces = [go.Scatter3d(\n            x=filtered_df['x'],\n            y=filtered_df['y'],\n            z=filtered_df['z'],\n            mode='markers',\n            marker=dict(\n                color=filtered_df.color,\n                size=3))]\n\n        for i, seg in enumerate(connections):\n            trace = go.Scatter3d(\n                    x=filtered_df.loc[seg]['x'],\n                    y=filtered_df.loc[seg]['y'],\n                    z=filtered_df.loc[seg]['z'],\n                    mode='lines',\n            )\n                # append the trace to the list of traces\n            traces.append(trace)\n        frame_data = go.Frame(data=traces, traces = [i for i in range(16)])\n        # append the frame to the list of frames\n        frames_l.append(frame_data)\n\n    traces = [go.Scatter3d(\n        x=first_frame_df['x'],\n        y=first_frame_df['y'],\n        z=first_frame_df['z'],\n        mode='markers',\n        marker=dict(\n            color=first_frame_df.color,\n            size=3\n        )\n    )]\n    for i, seg in enumerate(connections):\n        trace = go.Scatter3d(\n            x=first_frame_df.loc[seg]['x'],\n            y=first_frame_df.loc[seg]['y'],\n            z=first_frame_df.loc[seg]['z'],\n            mode='lines',\n            line=dict(\n                color='black',\n                width=3\n            )\n        )\n        # append the trace to the list of traces\n        traces.append(trace)\n    fig = go.Figure(\n        data=traces,\n        frames=frames_l\n    )\n\n    # Layout\n    fig.update_layout(\n        title=\"3D\",\n        width=500,\n        height=800,\n        scene={\n            'aspectmode': 'data',\n        },\n        updatemenus=[{\"buttons\": [\n                    {\"args\": [None, {\"frame\": {\"duration\": 100,\"redraw\": True},\n                                    \"fromcurrent\": True,\n                                    \"transition\": {\"duration\": 0}}],\n                        \"label\": \"&#9654;\",  # play symbol\n                        \"method\": \"animate\",},],\n                \"direction\": \"left\",\n                \"pad\": {\"r\": 100, \"t\": 100},\n                \"font\": {\"size\":16},\n                \"type\": \"buttons\",\n                \"x\": 0.1,\n                \"y\": 0,\n            }])\n    \n    camera = dict(\n        up=dict(x=0, y=-1, z=0),\n        eye=dict(x=0, y=0, z=2.5))\n    \n    fig.update_layout(scene_camera=camera, showlegend=False)\n    fig.update_layout(scene = dict(xaxis = dict(visible=False),\n            yaxis = dict(visible=False),\n            zaxis =dict(visible=False, autorange = \"reversed\")))\n    fig.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-07T14:07:42.764926Z","iopub.execute_input":"2023-03-07T14:07:42.765438Z","iopub.status.idle":"2023-03-07T14:07:42.791999Z","shell.execute_reply.started":"2023-03-07T14:07:42.765380Z","shell.execute_reply":"2023-03-07T14:07:42.790535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## DEMO","metadata":{}},{"cell_type":"code","source":"parquet_df = pd.read_parquet(\"/kaggle/input/asl-signs/train_landmark_files/26734/1000035562.parquet\")\nvisualise3d_landmarks(parquet_df)\n### YES ITS THAT EASY TO USE!!!\n### You can rotate, zoom and move the 3d plot below as much as you want! dont forget to press the play button!","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2023-03-07T14:07:43.844857Z","iopub.execute_input":"2023-03-07T14:07:43.845897Z","iopub.status.idle":"2023-03-07T14:07:46.514233Z","shell.execute_reply.started":"2023-03-07T14:07:43.845854Z","shell.execute_reply":"2023-03-07T14:07:46.513343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2D Version","metadata":{}},{"cell_type":"code","source":"def visualise2d_landmarks(parquet_df):\n    connections = [  \n        [0, 1, 2, 3, 4,],\n        [0, 5, 6, 7, 8],\n        [0, 9, 10, 11, 12],\n        [0, 13, 14, 15, 16],\n        [0, 17, 18, 19, 20],\n\n        \n        [38, 36, 35, 34, 30, 31, 32, 33, 37],\n        [40, 39],\n        [52, 46, 50, 48, 46, 44, 42, 41, 43, 45, 47, 49, 45, 51],\n        [42, 54, 56, 58, 60, 62, 58],\n        [41, 53, 55, 57, 59, 61, 57],\n        [54, 53],\n\n        \n        [80, 81, 82, 83, 84, ],\n        [80, 85, 86, 87, 88],\n        [80, 89, 90, 91, 92],\n        [80, 93, 94, 95, 96],\n        [80, 97, 98, 99, 100], ]\n\n\n    frames = sorted(set(parquet_df.frame))\n    first_frame = min(frames)\n    parquet_df['color'] = parquet_df.type.apply(lambda row: assign_color(row))\n    parquet_df['plot_order'] = parquet_df.apply(lambda row: assign_order(row), axis=1)\n    first_frame_df = parquet_df[parquet_df.frame == first_frame].copy()\n    first_frame_df = first_frame_df.sort_values([\"plot_order\"]).set_index('plot_order')\n\n\n    frames_l = []\n    for frame in frames:\n        filtered_df = parquet_df[parquet_df.frame == frame].copy()\n        filtered_df = filtered_df.sort_values([\"plot_order\"]).set_index(\"plot_order\")\n        traces = [go.Scatter(\n            x=filtered_df['x'],\n            y=filtered_df['y'],\n            mode='markers',\n            marker=dict(\n                color=filtered_df.color,\n                size=9))]\n\n        for i, seg in enumerate(connections):\n            trace = go.Scatter(\n                    x=filtered_df.loc[seg]['x'],\n                    y=filtered_df.loc[seg]['y'],\n                    mode='lines',\n            )\n            traces.append(trace)\n        frame_data = go.Frame(data=traces, traces = [i for i in range(17)])\n        frames_l.append(frame_data)\n\n    traces = [go.Scatter(\n        x=first_frame_df['x'],\n        y=first_frame_df['y'],\n        mode='markers',\n        marker=dict(\n            color=first_frame_df.color,\n            size=9\n        )\n    )]\n    for i, seg in enumerate(connections):\n        trace = go.Scatter(\n            x=first_frame_df.loc[seg]['x'],\n            y=first_frame_df.loc[seg]['y'],\n            mode='lines',\n            line=dict(\n                color='black',\n                width=2\n            )\n        )\n        traces.append(trace)\n    fig = go.Figure(\n        data=traces,\n        frames=frames_l\n    )\n\n\n    fig.update_layout(\n        title=\"2D\",\n        width=500,\n        height=800,\n        scene={\n            'aspectmode': 'data',\n        },\n        updatemenus=[{\"buttons\": [\n                    {\"args\": [None, {\"frame\": {\"duration\": 100,\"redraw\": True},\n                                    \"fromcurrent\": True,\n                                    \"transition\": {\"duration\": 0}}],\n                        \"label\": \"&#9654;\",  # play symbol\n                        \"method\": \"animate\",},],\n                \"direction\": \"left\",\n                \"pad\": {\"r\": 100, \"t\": 100},\n                \"font\": {\"size\":16},\n                \"type\": \"buttons\",\n                \"x\": 0.1,\n                \"y\": 0,\n            }])\n    camera = dict(\n        up=dict(x=0, y=-1, z=0),\n        eye=dict(x=0, y=0, z=2.5)\n    )\n    fig.update_layout(scene_camera=camera, showlegend=False)\n    fig.update_layout(xaxis = dict(visible=False),\n            yaxis = dict(visible=False),\n    )\n    fig.update_yaxes(autorange=\"reversed\")\n\n    fig.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-07T14:09:23.200946Z","iopub.execute_input":"2023-03-07T14:09:23.201333Z","iopub.status.idle":"2023-03-07T14:09:23.223444Z","shell.execute_reply.started":"2023-03-07T14:09:23.201294Z","shell.execute_reply":"2023-03-07T14:09:23.221829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualise2d_landmarks(parquet_df)","metadata":{"execution":{"iopub.status.busy":"2023-03-07T14:09:24.192244Z","iopub.execute_input":"2023-03-07T14:09:24.192642Z","iopub.status.idle":"2023-03-07T14:09:26.464525Z","shell.execute_reply.started":"2023-03-07T14:09:24.192610Z","shell.execute_reply":"2023-03-07T14:09:26.463577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}