{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🔎Unlock the Secrets of ASL: Interactive Dashboard using Plotly!","metadata":{}},{"cell_type":"markdown","source":"This notebook presents an interactive dashboard that allows exploring American Sign Language (ASL) landmarks in 2D space. By selecting different ASL categories and IDs, users can filter the displayed landmarks and study the hand shapes and movements associated with specific signs. The dashboard is built with Plotly and Python, and can be used for educational purposes or to support the development of ASL recognition systems. \n\nWith just a few lines of code, the notebook will generate a link to the dashboard, and you can begin exploring any ASL gestures given in the Google - Isolated Sign Language Recognition Dataset. Whether you're a student of ASL or simply interested in exploring this fascinating language, this tool is both easy to use and incredibly informative. The intuitive interface makes it easy to explore the intricacies of ASL gestures, gain a deeper understanding of this rich language, and of course help you fine tune the model by identifying what caused the drop in accuracy for a certain category.\n\n**Multi-page dashboard:**\n1. **2D Animation**\n2. **Mean Frames**\n\nSNEAK PEAK (🚨Watch All the Recording⚠️):\n\n**You only have to run the notebook and click on the second link at the end of the notebook!**","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"from IPython.display import HTML\nHTML('<div align=\"center\"><iframe align=\"middle\" width=\"1000\" height=\"904\" src=\"https://www.youtube.com/embed/LIT8zOhnIxs\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen></iframe></div>')","metadata":{"execution":{"iopub.status.busy":"2023-03-21T13:35:30.940448Z","iopub.execute_input":"2023-03-21T13:35:30.940920Z","iopub.status.idle":"2023-03-21T13:35:30.987109Z","shell.execute_reply.started":"2023-03-21T13:35:30.940874Z","shell.execute_reply":"2023-03-21T13:35:30.985848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Packages","metadata":{}},{"cell_type":"code","source":"import plotly.io as pio\npio.templates.default = \"simple_white\"\nimport plotly.graph_objects as go\n!pip install -q jupyter_dash\n!pip install dash-cytoscape -q\nimport dash_cytoscape as cyto \n\ntry:\n    from pyngrok import ngrok\nexcept:\n    !pip install -q pyngrok\n    from pyngrok import ngrok\n    \ntry:\n    import dash_bootstrap_components as dbc\nexcept:\n    !pip install -q dash_bootstrap_components\n    import dash_bootstrap_components as dbc\n\nfrom dash import dcc, dash\nimport warnings\n\nwarnings.filterwarnings(\"ignore\")\nfrom dash import html, dcc\nfrom dash.dependencies import Input, Output\nimport pandas as pd\nfrom jupyter_dash import JupyterDash \n\n!ngrok authtoken 2N6MyaoUfK1C0hvwSAajUzgLURx_yREPXyNkRz9Mv13x15Ba\ntunnel = ngrok.connect(8050)","metadata":{"execution":{"iopub.status.busy":"2023-03-22T14:54:20.139716Z","iopub.execute_input":"2023-03-22T14:54:20.140861Z","iopub.status.idle":"2023-03-22T14:55:09.491391Z","shell.execute_reply.started":"2023-03-22T14:54:20.140805Z","shell.execute_reply":"2023-03-22T14:55:09.489768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helper Functions","metadata":{}},{"cell_type":"code","source":"def assign_color(row):\n    if row == 'face':\n        return 'red'\n    elif 'hand' in row:\n        return 'dodgerblue'\n    else:\n        return 'green'\n\n\ndef assign_order(row):\n    if row.type == 'face':\n        return row.landmark_index + 101\n    elif row.type == 'pose':\n        return row.landmark_index + 30\n    elif row.type == 'left_hand':\n        return row.landmark_index + 80\n    else:\n        return row.landmark_index\n\ndef filter_group(group):\n    if group[group.type == 'right_hand'].x.isnull().all():\n        new_group = group[group.type != 'right_hand']\n        if new_group[new_group.type == 'left_hand'].x.isnull().all():\n             return pd.DataFrame(columns=group.columns)\n        return group\n    else:\n        new_group = group[group.type != 'left_hand']\n        if new_group[new_group.type == 'right_hand'].x.isnull().all():\n             return pd.DataFrame(columns=group.columns)\n\n\ndef visualise2d_landmarks(parquet_df):\n    connections = [  # right hand\n        [0, 1, 2, 3, 4,],\n        [0, 5, 6, 7, 8],\n        [0, 9, 10, 11, 12],\n        [0, 13, 14, 15, 16],\n        [0, 17, 18, 19, 20],\n\n        # pose\n        [38, 36, 35, 34, 30, 31, 32, 33, 37],\n        [40, 39],\n        [52, 46, 50, 48, 46, 44, 42, 41, 43, 45, 47, 49, 45, 51],\n        [42, 54, 56, 58, 60, 62, 58],\n        [41, 53, 55, 57, 59, 61, 57],\n        [54, 53],\n\n        # left hand\n        [80, 81, 82, 83, 84, ],\n        [80, 85, 86, 87, 88],\n        [80, 89, 90, 91, 92],\n        [80, 93, 94, 95, 96],\n        [80, 97, 98, 99, 100], ]\n\n\n    frames = sorted(set(parquet_df.frame))\n    first_frame = min(frames)\n    parquet_df['color'] = parquet_df.type.apply(lambda row: assign_color(row))\n    parquet_df['plot_order'] = parquet_df.apply(lambda row: assign_order(row), axis=1)\n    first_frame_df = parquet_df[parquet_df.frame == first_frame].copy()\n    first_frame_df = first_frame_df.sort_values([\"plot_order\"]).set_index('plot_order')\n\n\n    frames_l = []\n    for frame in frames:\n        filtered_df = parquet_df[parquet_df.frame == frame].copy()\n        filtered_df = filtered_df.sort_values([\"plot_order\"]).set_index(\"plot_order\")\n        traces = [go.Scatter(\n            x=filtered_df['x'],\n            y=filtered_df['y'],\n            # z=filtered_df['z'],\n            mode='markers',\n            marker=dict(\n                color=filtered_df.color,\n                size=9))]\n\n        for i, seg in enumerate(connections):\n            trace = go.Scatter(\n                    x=filtered_df.loc[seg]['x'],\n                    y=filtered_df.loc[seg]['y'],\n                    # z=filtered_df.loc[seg]['z'],\n                    mode='lines',\n            )\n            traces.append(trace)\n        frame_data = go.Frame(data=traces, traces = [i for i in range(17)])\n        frames_l.append(frame_data)\n\n    traces = [go.Scatter(\n        x=first_frame_df['x'],\n        y=first_frame_df['y'],\n        # z=first_frame_df['z'],\n        mode='markers',\n        marker=dict(\n            color=first_frame_df.color,\n            size=9\n        )\n    )]\n    for i, seg in enumerate(connections):\n        trace = go.Scatter(\n            x=first_frame_df.loc[seg]['x'],\n            y=first_frame_df.loc[seg]['y'],\n            # z=first_frame_df.loc[seg]['z'],\n            mode='lines',\n            line=dict(\n                color='black',\n                width=2\n            )\n        )\n        traces.append(trace)\n    fig = go.Figure(\n        data=traces,\n        frames=frames_l\n    )\n\n    # Layout\n    fig.update_layout(\n        title=\"ASL Sign Visualization\",\n        width=1000,\n        height=1800,\n        scene={\n            'aspectmode': 'data',\n        },\n        updatemenus=[\n            {\n                \"buttons\": [\n                    {\n                        \"args\": [None, {\"frame\": {\"duration\": 100,\n                                                  \"redraw\": True},\n                                        \"fromcurrent\": True,\n                                        \"transition\": {\"duration\": 0}}],\n                        \"label\": \"&#9654;\",\n                        \"method\": \"animate\",\n                    },\n\n                ],\n                \"direction\": \"left\",\n                \"pad\": {\"r\": 100, \"t\": 100},\n                \"font\": {\"size\":30},\n                \"type\": \"buttons\",\n                \"x\": 0.1,\n                \"y\": 0,\n            }\n        ],\n    )\n    camera = dict(\n        up=dict(x=0, y=-1, z=0),\n        eye=dict(x=0, y=0, z=2.5)\n    )\n    fig.update_layout(scene_camera=camera, showlegend=False)\n    fig.update_layout(xaxis = dict(visible=False),\n            yaxis = dict(visible=False),\n    )\n    fig.update_yaxes(autorange=\"reversed\")\n\n    # fig.show()\n    return fig\n\ndef visualiseavg_landmarks(parquet_df):\n    connections = [  # right hand\n        [0, 1, 2, 3, 4,],\n        [0, 5, 6, 7, 8],\n        [0, 9, 10, 11, 12],\n        [0, 13, 14, 15, 16],\n        [0, 17, 18, 19, 20],\n\n        # pose\n        [38, 36, 35, 34, 30, 31, 32, 33, 37],\n        [40, 39],\n        [52, 46, 50, 48, 46, 44, 42, 41, 43, 45, 47, 49, 45, 51],\n        [42, 54, 56, 58, 60, 62, 58],\n        [41, 53, 55, 57, 59, 61, 57],\n        [54, 53],\n\n        # left hand\n        [80, 81, 82, 83, 84, ],\n        [80, 85, 86, 87, 88],\n        [80, 89, 90, 91, 92],\n        [80, 93, 94, 95, 96],\n        [80, 97, 98, 99, 100], ]\n\n    first_frame_df = parquet_df.copy()\n    first_frame_df['color'] = first_frame_df.type.apply(lambda row: assign_color(row))\n    first_frame_df['plot_order'] = first_frame_df.apply(lambda row: assign_order(row), axis=1)\n    first_frame_df = first_frame_df.groupby(\"plot_order\").agg({'x':'mean', 'y':'mean', 'color':'first'}).reset_index()\n    first_frame_df = first_frame_df.sort_values([\"plot_order\"]).set_index('plot_order')\n\n    traces = [go.Scatter(\n        x=first_frame_df['x'],\n        y=first_frame_df['y'],\n        # z=first_frame_df['z'],\n        mode='markers',\n        marker=dict(\n            color=first_frame_df.color,\n            size=9\n        )\n    )]\n    for i, seg in enumerate(connections):\n        trace = go.Scatter(\n            x=first_frame_df.loc[seg]['x'],\n            y=first_frame_df.loc[seg]['y'],\n            # z=first_frame_df.loc[seg]['z'],\n            mode='lines',\n            line=dict(\n                color='black',\n                width=2\n            )\n        )\n        traces.append(trace)\n    fig = go.Figure(\n        data=traces,\n    )\n\n    # Layout\n    fig.update_layout(\n        title=\"ASL Sign Visualization\",\n        width=1000,\n        height=1800)\n    fig.update_layout(showlegend=False)\n    fig.update_layout(xaxis = dict(visible=False),\n                      yaxis = dict(visible=False),\n                      )\n    fig.update_yaxes(autorange=\"reversed\")\n\n    return fig","metadata":{"execution":{"iopub.status.busy":"2023-03-22T14:55:09.493736Z","iopub.execute_input":"2023-03-22T14:55:09.494058Z","iopub.status.idle":"2023-03-22T14:55:09.530355Z","shell.execute_reply.started":"2023-03-22T14:55:09.494028Z","shell.execute_reply":"2023-03-22T14:55:09.528101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## App & Navigation bar","metadata":{}},{"cell_type":"code","source":"app = JupyterDash(__name__, server_url=tunnel.public_url, meta_tags=[{\"name\": \"viewport\", \"content\": \"width=device-width\"}],\n                external_stylesheets=[dbc.themes.BOOTSTRAP],suppress_callback_exceptions=True)\n\ndef Navbar():\n    layout = html.Div([\n        dbc.NavbarSimple(\n            children=[\n                dbc.NavItem(dbc.NavLink(\"2D Visuals\", href=\"/landmark2d\", id=\"landmark2d\")),\n                dbc.NavItem(dbc.NavLink(\"Mean Visual\", href=\"/landmarkavg\", id=\"landmarkavg\")),\n            ] ,\n            brand=\"Landmark 2D Visualization Dashboard\",\n            brand_href=\"/landmark2d\",\n            id=\"navbar\",\n            color=\"dark\",\n            dark=True,\n        ),\n    ])\n\n    return layout\n\ndata_directory = '/kaggle/input/asl-signs/'\ntrain_data = pd.read_csv(data_directory + 'train.csv')\ndropdown_options = train_data.groupby('sign')['sequence_id'].apply(list).to_dict()\nnames = list(dropdown_options.keys())\nnestedOptions = dropdown_options[names[0]]","metadata":{"execution":{"iopub.status.busy":"2023-03-22T14:55:09.531690Z","iopub.execute_input":"2023-03-22T14:55:09.533110Z","iopub.status.idle":"2023-03-22T14:55:09.959431Z","shell.execute_reply.started":"2023-03-22T14:55:09.533072Z","shell.execute_reply":"2023-03-22T14:55:09.957141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2D Landmarks Video Page","metadata":{}},{"cell_type":"code","source":"def visual_dash(sequence_id):\n    data = train_data.copy()\n    parquet_file = data[data.sequence_id == sequence_id]\n    parquet_path = parquet_file.path.values[0]\n    sign_cat = parquet_file.sign.values[0]\n\n    parquet_df = pd.read_parquet(data_directory + parquet_path)\n    fig = visualise2d_landmarks(parquet_df)\n\n    return fig, sign_cat\n\nfig, sign_cat = visual_dash(1002052130)\n\n\nsign_dropdown1 = dcc.Dropdown(options=[{'label':name, 'value':name} for name in names],\n                                  id='sign_dropdown',\n                                  clearable=False,\n                                  value = names[0], className=\"dbc\",\n                                  placeholder='Select a Sign', maxHeight=200)\n\nsequence_dropdown1 = dcc.Dropdown(id='sequence_dropdown',\n                               clearable=False,\n                               className=\"dbc\",\n                               placeholder='Select a Sequence ID', maxHeight=200)\n\nlayout_2d = dbc.Container(\n    [dbc.Row([dbc.Col(sign_dropdown1),\n              dbc.Col(sequence_dropdown1),]),\n     dbc.Row([html.H1(id='sign_cat'),\n         dbc.Col([\n             dcc.Graph(id='fig', figure=fig,\n                       style={'height': 2000}),\n             html.Hr()\n         ], width={'size': 12, 'offset': 0, 'order': 1})])]\n)","metadata":{"execution":{"iopub.status.busy":"2023-03-22T14:55:09.961565Z","iopub.execute_input":"2023-03-22T14:55:09.961899Z","iopub.status.idle":"2023-03-22T14:55:21.396858Z","shell.execute_reply.started":"2023-03-22T14:55:09.961856Z","shell.execute_reply":"2023-03-22T14:55:21.394821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Mean Frame Page","metadata":{}},{"cell_type":"code","source":"def visualmean_dash(sequence_id):\n    data = train_data.copy()\n    parquet_file = data[data.sequence_id == sequence_id]\n    parquet_path = parquet_file.path.values[0]\n    sign_cat = parquet_file.sign.values[0]\n\n    parquet_df = pd.read_parquet(data_directory + parquet_path)\n    fig = visualiseavg_landmarks(parquet_df)\n\n    return fig, sign_cat\n\nfig, sign_cat = visualmean_dash(1187993400)\n\n\nsign_dropdown = dcc.Dropdown(options=[{'label':name, 'value':name} for name in names],\n                             id='sign_dropdown',\n                             clearable=False,\n                             value = names[0], className=\"dbc\",\n                             placeholder='Select a Sign', maxHeight=200)\n\nsequence_dropdown = dcc.Dropdown(id='sequence_dropdown',\n                                 clearable=False,\n                                 className=\"dbc\",\n                                 placeholder='Select a Sequence ID', maxHeight=200)\n\nlayout_mean = dbc.Container(\n    [dbc.Row([dbc.Col(sign_dropdown),\n              dbc.Col(sequence_dropdown),]),\n     dbc.Row([html.H1(id='sign_cat2'),\n              dbc.Col([\n                  dcc.Graph(id='fig2', figure=fig,\n                            style={'height': 2000}),\n                  html.Hr()\n              ], width={'size': 12, 'offset': 0, 'order': 1})])]\n)","metadata":{"execution":{"iopub.status.busy":"2023-03-22T14:55:21.398480Z","iopub.execute_input":"2023-03-22T14:55:21.398769Z","iopub.status.idle":"2023-03-22T14:55:21.653933Z","shell.execute_reply.started":"2023-03-22T14:55:21.398741Z","shell.execute_reply":"2023-03-22T14:55:21.652497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Putting Everything Together!","metadata":{}},{"cell_type":"code","source":"nav = Navbar()\n\napp.layout = html.Div([\n    dcc.Location(id='url', refresh=False),\n    nav,\n    html.Div(id='page-content', children=[]),\n])\n\n@app.callback(Output('page-content', 'children'),\n              [Input('url', 'pathname')])\n\ndef display_page(pathname):\n    if pathname == '/landmark2d':\n        return layout_2d\n    if pathname == '/landmarkavg':\n        return layout_mean\n    else:\n        return layout_2d\n\n@app.callback(\n     Output('sequence_dropdown', 'options'),\n    [Input('sign_dropdown', 'value')]\n)\ndef update_dropdown(name):\n    return [{'label': i, 'value': i} for i in dropdown_options[name]]\n\n@app.callback([Output(component_id=\"fig\", component_property=\"figure\"),\n               Output(component_id=\"sign_cat\", component_property=\"children\")],\n              [Input('sequence_dropdown', 'value')])\ndef callback_function(sequence_id):\n    fig, sign_cat = visual_dash(sequence_id)\n    return fig, sign_cat\n\n@app.callback([Output(component_id=\"fig2\", component_property=\"figure\"),\n               Output(component_id=\"sign_cat2\", component_property=\"children\")],\n              [Input('sequence_dropdown', 'value')])\ndef callback_function(sequence_id):\n    fig, sign_cat = visualmean_dash(sequence_id)\n    return fig, sign_cat\n\n# Run the app on localhost:8050\napp.run_server(mode='external')","metadata":{"execution":{"iopub.status.busy":"2023-03-22T14:55:21.655253Z","iopub.execute_input":"2023-03-22T14:55:21.655631Z","iopub.status.idle":"2023-03-22T14:55:21.717849Z","shell.execute_reply.started":"2023-03-22T14:55:21.655596Z","shell.execute_reply":"2023-03-22T14:55:21.716731Z"},"trusted":true},"execution_count":null,"outputs":[]}]}