{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":46105,"databundleVersionId":5087314,"sourceType":"competition"}],"dockerImageVersionId":30407,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<div style=\"color:white; \n            background-color:white; \n            background-image:url('https://www.nicepng.com/png/full/392-3922851_28-collection-of-sign-language-clipart-pictures-sign.png');\n            background-position: center; \n            background-repeat: no-repeat; \n            background-size: cover;\n            height:300px;\">\n    <span style=\"color:white\">  g  </span>\n    <h1 style=\"color:black; font-size:4em; margin: 1% auto 1% 5%\">Isolated Sign Language Recognition</h1>\n    <h2 style=\"color:black; font-size:2em; margin: 5% auto 1% 7%\">Step 1 - Visualizing data</h2>\n    <span style=\"color:white\">    </span>\n</div>\n\n","metadata":{}},{"cell_type":"markdown","source":"**Other notebook**\n\nhttps://www.kaggle.com/code/cristaliss/islr-new-hands-only-dataset-preprocessing","metadata":{}},{"cell_type":"markdown","source":"# 1. Introduction\n## Sign Language Recognition\nSign language recognition is a field of research that seeks to enable machines to understand and interpret sign language. Sign language relies on gestures and facial expressions to convey meaning, and is used by people with hearing impairments to communicate. \n\nBy leveraging machine learning algorithms and computer vision techniques, researchers have been able to develop models that can recognize and interpret sign language. These models have a variety of applications in educational, healthcare, and other settings, and are helping to bridge the communication gap between the deaf and hearing communities.\n\n## Visualizing data\n\nVisualizing data is an essential part of training machine learning models. It allows us to gain insights into the data, identify patterns, and understand the structure of the data. Visualizing data can help us make decisions about the model architecture and hyperparameter settings, as well as provide feedback on the model's performance. Furthermore, data visualization can help us detect potential issues such as outliers, missing values, and class imbalance. By visualizing data before training, we can gain valuable insights into the underlying data and make informed decisions about the model's architecture, hyperparameters, and performance.","metadata":{}},{"cell_type":"markdown","source":"# 2. Defying constants, libraries and more","metadata":{}},{"cell_type":"markdown","source":"\nConstants and libraries are essential components of any Python program. Constants are variables whose value cannot be changed and libraries are collections of code that can be reused to perform specific tasks. \n\nIn Python, constants are usually declared and assigned in a module and libraries are initialized with the import statement. Using constants and libraries can help improve the readability and maintainability of a program, as well as help to ensure that code is being reused efficiently.","metadata":{}},{"cell_type":"code","source":"DATA_PATH = \"../input/asl-signs/\"","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:18:16.570133Z","iopub.execute_input":"2023-02-27T11:18:16.570705Z","iopub.status.idle":"2023-02-27T11:18:16.580758Z","shell.execute_reply.started":"2023-02-27T11:18:16.570669Z","shell.execute_reply":"2023-02-27T11:18:16.579545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os \nimport json\n\n# Visualize\nimport sys\nimport csv\nimport matplotlib.pyplot as plt\n\n#Animation\nfrom matplotlib import animation\nfrom IPython.display import HTML","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:19:12.058782Z","iopub.execute_input":"2023-02-27T11:19:12.059212Z","iopub.status.idle":"2023-02-27T11:19:12.065340Z","shell.execute_reply.started":"2023-02-27T11:19:12.059174Z","shell.execute_reply":"2023-02-27T11:19:12.064180Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Loading data\n\nBefore training a model, it is essential to first load the necessary data. This data may come from a variety of sources such as files, databases, or other external sources. In order to get the most out of the model, it is important to make sure that the data is of high quality and is in a format that is suitable for the model. This may involve pre-processing steps such as cleaning, normalizing, or transforming the data. Once the data is loaded, it can then be used to train the model by providing it with the necessary inputs and outputs. After the model is trained, it can then be tested against new data to ensure that it is performing as expected.\n\nIn this case, we are going to load and visualize *train.csv* file, which contains our mapping between the action occurred in a sequence (*sequence_id*) and the sign (*expected sign to predict*).\n\nThen, we will load the *sign_to_prediction_index_map.json*, which match a number to each type of sign.\nFinally, some *sequences* will be load. We will try to make a visual approach to them.","metadata":{}},{"cell_type":"code","source":"def read_dict(file_path):\n    path = os.path.expanduser(file_path)\n    with open(path, \"r\") as f:\n        dic = json.load(f)\n    return dic\n\ndef get_data(data_path):\n    train = pd.read_csv(f'{data_path}train.csv')\n    index_mapping = read_dict(f'{data_path}sign_to_prediction_index_map.json')\n    index_mapping = dict([(index_mapping[key], key) for key in index_mapping])\n    sequence_sample = data = pd.read_parquet(f'{data_path}train_landmark_files/26734/1000035562.parquet')\n    return train, index_mapping, sequence_sample","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:18:37.267337Z","iopub.execute_input":"2023-02-27T11:18:37.267728Z","iopub.status.idle":"2023-02-27T11:18:37.281248Z","shell.execute_reply.started":"2023-02-27T11:18:37.267693Z","shell.execute_reply":"2023-02-27T11:18:37.279913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train, index_mapping, sequence_sample = get_data(DATA_PATH)","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:18:37.734432Z","iopub.execute_input":"2023-02-27T11:18:37.735014Z","iopub.status.idle":"2023-02-27T11:18:37.913748Z","shell.execute_reply.started":"2023-02-27T11:18:37.734964Z","shell.execute_reply":"2023-02-27T11:18:37.912591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:18:38.733866Z","iopub.execute_input":"2023-02-27T11:18:38.734668Z","iopub.status.idle":"2023-02-27T11:18:38.750806Z","shell.execute_reply.started":"2023-02-27T11:18:38.734612Z","shell.execute_reply":"2023-02-27T11:18:38.749510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"index_mapping","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:18:39.217917Z","iopub.execute_input":"2023-02-27T11:18:39.218957Z","iopub.status.idle":"2023-02-27T11:18:39.233258Z","shell.execute_reply.started":"2023-02-27T11:18:39.218906Z","shell.execute_reply":"2023-02-27T11:18:39.231976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sequence_sample","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:18:39.806615Z","iopub.execute_input":"2023-02-27T11:18:39.807720Z","iopub.status.idle":"2023-02-27T11:18:39.830625Z","shell.execute_reply.started":"2023-02-27T11:18:39.807666Z","shell.execute_reply":"2023-02-27T11:18:39.829250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Visualizing sequence data\n\nNow, we have a problem! \n\nSequence of actions are not images we could plot. They are dots obtaing with MediaPipe Holistic pipeline by Google.\n\n\"The MediaPipe Holistic pipeline integrates separate models for pose, face and hand components, each of which are optimized for their particular domain. However, because of their different specializations, the input to one component is not well-suited for the others. The pose estimation model, for example, takes a lower, fixed resolution video frame (256x256) as input. But if one were to crop the hand and face regions from that image to pass to their respective models, the image resolution would be too low for accurate articulation. Therefore, we designed MediaPipe Holistic as a multi-stage pipeline, which treats the different regions using a region appropriate image resolution.\" (https://google.github.io/mediapipe/solutions/holistic.html)\n\n\nSo, seeing the data: ","metadata":{}},{"cell_type":"code","source":"sequence_sample","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:18:54.115190Z","iopub.execute_input":"2023-02-27T11:18:54.115883Z","iopub.status.idle":"2023-02-27T11:18:54.138318Z","shell.execute_reply.started":"2023-02-27T11:18:54.115844Z","shell.execute_reply":"2023-02-27T11:18:54.137119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we can see the different columns of the dataframe. The first one corresponds to the frame number, thus including the 'time' in the dataset. Next, a unique identifier is included to the row composed of different parameters. The 'type' column, in turn, refers to the body part to which the frame belongs. Finally, different x, y, z coordinates corresponding to a point in three-dimensional space are included.\n\n\nTo consider, within the same frame, we will find different points that will finally show us shapes to interpret in the three-dimensional space.\n\n\n\nLet's keep a frame of the face to analyse it more closely.","metadata":{}},{"cell_type":"code","source":"face_20 = sequence_sample.loc[(sequence_sample.frame == 20) & (sequence_sample.type == 'face')]\nface_20","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:19:00.585214Z","iopub.execute_input":"2023-02-27T11:19:00.585958Z","iopub.status.idle":"2023-02-27T11:19:00.608890Z","shell.execute_reply.started":"2023-02-27T11:19:00.585917Z","shell.execute_reply":"2023-02-27T11:19:00.607711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we are going to draw it in a flat 2-D space since the three-dimensional visualisation makes us lose some interpretation. The z-coordinate, corresponding to the depth, will be expressed by modifying the colour of the points we draw. The darker, the greater the depth, the lighter, the closer to us.","metadata":{}},{"cell_type":"code","source":"X, Y, Z = np.array(face_20.x), np.array(face_20.y), np.array(face_20.z)\n\n# Plot X,Y,Z\nfig = plt.figure(figsize=(5,5))\nax = fig.add_subplot()#projection='3d')\nax.scatter(X, -Y,c=Z, cmap=\"copper\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:19:16.800365Z","iopub.execute_input":"2023-02-27T11:19:16.801113Z","iopub.status.idle":"2023-02-27T11:19:17.025660Z","shell.execute_reply.started":"2023-02-27T11:19:16.801073Z","shell.execute_reply":"2023-02-27T11:19:17.024561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Animation\n\nSince a single frame tells us nothing beyond the shape of the figure to be displayed. Let's animate the sequence of figures and get an animation to interpret.\n\n\nTo begin with, let's visualise the animation of the face.","metadata":{}},{"cell_type":"code","source":"face = sequence_sample.loc[(sequence_sample.type == 'face')]\nface","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:21:07.573783Z","iopub.execute_input":"2023-02-27T11:21:07.574854Z","iopub.status.idle":"2023-02-27T11:21:07.596818Z","shell.execute_reply.started":"2023-02-27T11:21:07.574811Z","shell.execute_reply":"2023-02-27T11:21:07.595682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"frames_numbers = face.frame.unique()","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:21:11.352753Z","iopub.execute_input":"2023-02-27T11:21:11.353144Z","iopub.status.idle":"2023-02-27T11:21:11.359382Z","shell.execute_reply.started":"2023-02-27T11:21:11.353108Z","shell.execute_reply":"2023-02-27T11:21:11.358119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"frames = []\nfor frame_n in frames_numbers:\n    frames.append(face.loc[face.frame == frame_n])\n    ","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-02-27T11:21:17.595887Z","iopub.execute_input":"2023-02-27T11:21:17.596495Z","iopub.status.idle":"2023-02-27T11:21:17.614063Z","shell.execute_reply.started":"2023-02-27T11:21:17.596450Z","shell.execute_reply":"2023-02-27T11:21:17.613051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we are going to animate the frames, for this we use the FuncAnimation function that will allow us to print a different frame in each time step and in this way visualise the complete animation.","metadata":{}},{"cell_type":"code","source":"# First set up the figure, the axis, and the plot element we want to animate\nfig = plt.figure(figsize=(6,6))\nax = plt.axes(xlim=(.4, .7), ylim=(-.5, -.2))\nscat = ax.scatter([], [])\nline, = ax.plot([], [], lw=2)\n\n\n# initialization function: plot the background of each frame\ndef init():\n    line.set_data([], [])\n    return line,\n\n# animation function.  This is called sequentially\ndef animate(i):\n    x = np.array(i.x)\n    y = -np.array(i.y)\n    z = np.array(i.z)\n    scat.set_offsets(np.c_[x, y])\n    \n    cmap = plt.cm.copper\n    norm = plt.Normalize(vmin=min(z), vmax=max(z))\n    colors = cmap(norm(z))\n    scat.set_color(colors)\n    return scat,\n\n# call the animator.  blit=True means only re-draw the parts that have changed.\nanim = animation.FuncAnimation(fig, animate, init_func=init,\n                               frames=frames,interval=1, blit=True)\n\n\nHTML(anim.to_jshtml(fps=10))","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:27:05.121610Z","iopub.execute_input":"2023-02-27T11:27:05.122692Z","iopub.status.idle":"2023-02-27T11:27:07.555947Z","shell.execute_reply.started":"2023-02-27T11:27:05.122633Z","shell.execute_reply":"2023-02-27T11:27:07.554783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### DID YOU LIKE IT? LET´S ANIMATE MORE POINTS.","metadata":{}},{"cell_type":"code","source":"sequence_sample.type.unique()","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:29:15.263781Z","iopub.execute_input":"2023-02-27T11:29:15.264816Z","iopub.status.idle":"2023-02-27T11:29:15.273975Z","shell.execute_reply.started":"2023-02-27T11:29:15.264773Z","shell.execute_reply":"2023-02-27T11:29:15.272605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def animate_frames(data,xlim, ylim):\n    # First set up the figure, the axis, and the plot element we want to animate\n    fig = plt.figure(figsize=(6,6))\n    ax = plt.axes(xlim=xlim, ylim=ylim)\n    scat = ax.scatter([], [])\n    line, = ax.plot([], [], lw=2)\n\n\n    # initialization function: plot the background of each frame\n    def init():\n        line.set_data([], [])\n        return line,\n\n    # animation function.  This is called sequentially\n    def animate(i):\n        x = np.array(i.x)\n        y = -np.array(i.y)\n        z = np.array(i.z)\n        scat.set_offsets(np.c_[x, y])\n\n        cmap = plt.cm.copper\n        norm = plt.Normalize(vmin=min(z), vmax=max(z))\n        colors = cmap(norm(z))\n        scat.set_color(colors)\n        return scat,\n\n    # call the animator.  blit=True means only re-draw the parts that have changed.\n    anim = animation.FuncAnimation(fig, animate, init_func=init,\n                                   frames=data,interval=1, blit=True)\n\n\n    return HTML(anim.to_jshtml(fps=10))","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:39:05.939864Z","iopub.execute_input":"2023-02-27T11:39:05.940283Z","iopub.status.idle":"2023-02-27T11:39:05.951095Z","shell.execute_reply.started":"2023-02-27T11:39:05.940246Z","shell.execute_reply":"2023-02-27T11:39:05.949505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = sequence_sample.loc[(sequence_sample.type == 'right_hand')]\ndata.dropna(inplace=True)\nframes = []\nframes_numbers = data.frame.unique()\nfor frame_n in frames_numbers:\n    frames.append(data.loc[data.frame == frame_n])\nanimate_frames(frames,xlim=(-0.5, 1.0), ylim=(-.8, -.3))","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:43:08.933297Z","iopub.execute_input":"2023-02-27T11:43:08.933712Z","iopub.status.idle":"2023-02-27T11:43:09.864309Z","shell.execute_reply.started":"2023-02-27T11:43:08.933675Z","shell.execute_reply":"2023-02-27T11:43:09.862443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = sequence_sample\ndata.dropna(inplace=True)\nframes = []\nframes_numbers = data.frame.unique()\nfor frame_n in frames_numbers:\n    frames.append(data.loc[data.frame == frame_n])\nanimate_frames(frames,xlim=(-1, 2), ylim=(-2.5, 0))","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:44:15.731392Z","iopub.execute_input":"2023-02-27T11:44:15.732130Z","iopub.status.idle":"2023-02-27T11:44:17.976622Z","shell.execute_reply.started":"2023-02-27T11:44:15.732090Z","shell.execute_reply":"2023-02-27T11:44:17.970356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 7. Animating more data","metadata":{}},{"cell_type":"code","source":"sequence1 = data = pd.read_parquet(f'{DATA_PATH}train_landmark_files/16069/100015657.parquet')\nsequence2 = data = pd.read_parquet(f'{DATA_PATH}train_landmark_files/16069/1002113535.parquet')\nsequence3 = data = pd.read_parquet(f'{DATA_PATH}train_landmark_files/16069/10042041.parquet')\nsequence4 = data = pd.read_parquet(f'{DATA_PATH}train_landmark_files/16069/1004211348.parquet')\nsequence5 = data = pd.read_parquet(f'{DATA_PATH}train_landmark_files/16069/100438640.parquet')\nsequence6 = data = pd.read_parquet(f'{DATA_PATH}train_landmark_files/16069/1005009451.parquet')\nsequence7 = data = pd.read_parquet(f'{DATA_PATH}train_landmark_files/16069/1005223850.parquet')\nsequence8 = data = pd.read_parquet(f'{DATA_PATH}train_landmark_files/16069/1005492440.parquet')","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:50:38.594962Z","iopub.execute_input":"2023-02-27T11:50:38.595675Z","iopub.status.idle":"2023-02-27T11:50:38.773176Z","shell.execute_reply.started":"2023-02-27T11:50:38.595636Z","shell.execute_reply":"2023-02-27T11:50:38.772106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sequences = [sequence1,sequence2,sequence3,sequence4,sequence5,sequence6,sequence7,sequence8]","metadata":{"execution":{"iopub.status.busy":"2023-02-27T11:50:39.494457Z","iopub.execute_input":"2023-02-27T11:50:39.495553Z","iopub.status.idle":"2023-02-27T11:50:39.500704Z","shell.execute_reply.started":"2023-02-27T11:50:39.495475Z","shell.execute_reply":"2023-02-27T11:50:39.499613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for sequence in sequences:\n    hand = sequence.loc[(sequence.type == 'left_hand')]\n    face = sequence.loc[(sequence.type == 'face')]\n\n    sequence = pd.concat([hand,face])\n    sequence.dropna(inplace=True)\n    frames = []\n    frames_numbers = sequence.frame.unique()\n    for frame_n in frames_numbers:\n        frames.append(sequence.loc[sequence.frame == frame_n])\n    ani = animate_frames(frames,xlim=(0, 1.5), ylim=(-1, 0))\n    display(ani)","metadata":{"execution":{"iopub.status.busy":"2023-02-27T12:06:20.147298Z","iopub.execute_input":"2023-02-27T12:06:20.148658Z","iopub.status.idle":"2023-02-27T12:07:09.475238Z","shell.execute_reply.started":"2023-02-27T12:06:20.148603Z","shell.execute_reply":"2023-02-27T12:07:09.474022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 8. LAST BUT NOT LEAST: BYEE!\n\nI wish you'd find this notebook useful!!!!\n\n**REMEMBER TO COMMENT IF YOU WISH AND UP!**","metadata":{}}]}