{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n ","metadata":{}},{"cell_type":"markdown","source":"👏   IF YOU FORK THIS OR FIND THIS HELPFUL   👏\n\n**PLEASE UPVOTE!**\n\n\nTOC\n\n* [1. Background Knowledge](#section-one)\n    - [What is American Sign Language (ASL)](#subsection-one)\n    - [MediaPipe Holistic Package](#subsection-two)\n* [2. Landmarks Visualization](#section-two)","metadata":{}},{"cell_type":"markdown","source":"<a id='section-one'></a>\n## 1.Background Knowledge \n<a id='subsection-one'></a>\n### What is American Sign Language (ASL)\n\nAmerican Sign Language (ASL) is a complete, natural language that is used by many deaf and hard-of-hearing individuals in the United States and Canada. It is a visual language that uses hand gestures, facial expressions, and body movements to convey meaning.background knowledge\n\nASL has its own grammar and syntax, and is not simply a visual representation of English. It has a rich vocabulary and can express complex ideas and concepts. ASL is not universal, and different countries have their own sign languages with unique features.\n\nLearning ASL can provide a means of communication for those who are deaf or hard-of-hearing, as well as a better understanding and appreciation for Deaf culture. ASL interpretation is also an important profession, providing accessibility for deaf individuals in various settings, such as schools, workplaces, and public events.\n\nASL has its own unique grammar and syntax that make it a distinct language with its own rules and conventions.\n\ndemo video dowlond from https://www.aslpro.cc/ (American Sign Language Video Dictionaries)  This website is very good,recommended to visit","metadata":{}},{"cell_type":"code","source":"from IPython.display import HTML \nfrom base64 import b64encode\n\n# from :https://stackoverflow.com/questions/18019477/how-can-i-play-a-local-video-in-my-ipython-notebook\ndef display_video(path):  \n    with open(path,'rb') as f:\n        mp4 = f.read()   \n        data_url = f\"data:video/mp4;base64,\" + b64encode(mp4).decode()\n    display(\n      HTML(\n      \"\"\"\n          <video width=400 controls>\n                <source src=\"%s\" type=\"video/mp4\">\n          </video>\n      \"\"\" % data_url\n           )   \n    )","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:39:38.07608Z","iopub.execute_input":"2023-03-05T13:39:38.076643Z","iopub.status.idle":"2023-03-05T13:39:38.111637Z","shell.execute_reply.started":"2023-03-05T13:39:38.076588Z","shell.execute_reply":"2023-03-05T13:39:38.110633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_video('/kaggle/input/asl-demo-from-aslpro/tv.mp4') ","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:39:39.416394Z","iopub.execute_input":"2023-03-05T13:39:39.416825Z","iopub.status.idle":"2023-03-05T13:39:39.446129Z","shell.execute_reply.started":"2023-03-05T13:39:39.416785Z","shell.execute_reply":"2023-03-05T13:39:39.445009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_video('/kaggle/input/asl-demo-from-aslpro/after.mp4') ","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:39:41.164744Z","iopub.execute_input":"2023-03-05T13:39:41.1652Z","iopub.status.idle":"2023-03-05T13:39:41.181697Z","shell.execute_reply.started":"2023-03-05T13:39:41.165156Z","shell.execute_reply":"2023-03-05T13:39:41.180419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" <a id='subsection-two'></a>\n ### MediaPipe Holistic Package\n \nlandmark data extracted using the MediaPipe Holistic Solution. We can use it to see how it works, or how did the data come from.\n \n https://google.github.io/mediapipe/solutions/holistic.html","metadata":{}},{"cell_type":"code","source":"!pip install mediapipe","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:39:43.529151Z","iopub.execute_input":"2023-03-05T13:39:43.529538Z","iopub.status.idle":"2023-03-05T13:40:00.699725Z","shell.execute_reply.started":"2023-03-05T13:39:43.529505Z","shell.execute_reply":"2023-03-05T13:40:00.698152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import cv2\nimport mediapipe as mp","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:40:00.702238Z","iopub.execute_input":"2023-03-05T13:40:00.702659Z","iopub.status.idle":"2023-03-05T13:40:00.986543Z","shell.execute_reply.started":"2023-03-05T13:40:00.702614Z","shell.execute_reply":"2023-03-05T13:40:00.985184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_landmarks(filename):\n    cap = cv2.VideoCapture()\n    cap.open(f'/kaggle/input/asl-demo-from-aslpro/{filename}.mp4')\n    w = cap.get(cv2.CAP_PROP_FRAME_WIDTH)\n    h = cap.get(cv2.CAP_PROP_FRAME_HEIGHT)\n    count = cap.get(cv2.CAP_PROP_FRAME_COUNT)\n    fps = cap.get(cv2.CAP_PROP_FPS)\n    print('w: {}, h: {}, count: {}, fps: {}'.format(w, h, count, fps))\n    fourcc = cv2.VideoWriter_fourcc(*'mp4v')\n    out = cv2.VideoWriter(f'{filename}_save.mp4', fourcc, fps, (int(w), int(h)), True)\n    while cap.isOpened():\n        ret, frame = cap.read()\n        if ret == False:\n            break\n        frame = cv2.resize(frame, (int(w), int(h)), interpolation=cv2.INTER_LINEAR)\n        holistic = mp.solutions.holistic.Holistic()\n        results = holistic.process(frame)\n        mp_drawing = mp.solutions.drawing_utils\n\n        if results.pose_landmarks is not None:\n            mp_drawing.draw_landmarks(frame, results.pose_landmarks, mp.solutions.holistic.POSE_CONNECTIONS)\n        if results.face_landmarks is not None:\n            mp_drawing.draw_landmarks(frame, results.face_landmarks, mp.solutions.holistic.FACEMESH_CONTOURS)\n        if results.right_hand_landmarks is not None:\n            mp_drawing.draw_landmarks(frame, results.right_hand_landmarks, mp.solutions.holistic.HAND_CONNECTIONS)\n        if results.left_hand_landmarks is not None:\n            mp_drawing.draw_landmarks(frame, results.left_hand_landmarks, mp.solutions.holistic.HAND_CONNECTIONS)\n        out.write(frame)\n    cap.release()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:40:00.98814Z","iopub.execute_input":"2023-03-05T13:40:00.988527Z","iopub.status.idle":"2023-03-05T13:40:01.002319Z","shell.execute_reply.started":"2023-03-05T13:40:00.988489Z","shell.execute_reply":"2023-03-05T13:40:00.99966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_landmarks('after')\n","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:40:01.006208Z","iopub.execute_input":"2023-03-05T13:40:01.006705Z","iopub.status.idle":"2023-03-05T13:40:25.522427Z","shell.execute_reply.started":"2023-03-05T13:40:01.006581Z","shell.execute_reply":"2023-03-05T13:40:25.521209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# convert it to H.264  for webbrowser \n# from https://stackoverflow.com/questions/69294075/how-can-i-play-video-or-audio-on-a-jupyter-notebook-through-vs-code\n! ffmpeg -i /kaggle/working/after_save.mp4 /kaggle/working/after_save1.mp4\ndisplay_video('/kaggle/working/after_save1.mp4')","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:43:06.775848Z","iopub.execute_input":"2023-03-05T13:43:06.776346Z","iopub.status.idle":"2023-03-05T13:43:08.393973Z","shell.execute_reply.started":"2023-03-05T13:43:06.776303Z","shell.execute_reply":"2023-03-05T13:43:08.392557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_landmarks('tv')\n! ffmpeg -i /kaggle/working/tv_save.mp4 /kaggle/working/tv_save1.mp4\ndisplay_video('/kaggle/working/tv_save1.mp4')","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:45:00.478467Z","iopub.execute_input":"2023-03-05T13:45:00.47892Z","iopub.status.idle":"2023-03-05T13:45:32.235576Z","shell.execute_reply.started":"2023-03-05T13:45:00.478882Z","shell.execute_reply":"2023-03-05T13:45:32.233383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='section-two'></a>\n## 2 Landmarks Visualization","metadata":{}},{"cell_type":"code","source":"#pyarrow\n#!pip install pyarrow","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:48:32.949577Z","iopub.execute_input":"2023-03-05T13:48:32.950113Z","iopub.status.idle":"2023-03-05T13:48:46.854065Z","shell.execute_reply.started":"2023-03-05T13:48:32.950072Z","shell.execute_reply":"2023-03-05T13:48:46.852652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom  matplotlib import pyplot as plt\nimport mediapipe as mp\nfrom mediapipe.framework.formats import landmark_pb2\nimport cv2\nmp_holistic = mp.solutions.holistic","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:48:47.629995Z","iopub.execute_input":"2023-03-05T13:48:47.630959Z","iopub.status.idle":"2023-03-05T13:48:47.637413Z","shell.execute_reply.started":"2023-03-05T13:48:47.630886Z","shell.execute_reply":"2023-03-05T13:48:47.636002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df=pd.read_csv('/kaggle/input/asl-signs/train.csv')\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:49:15.092428Z","iopub.execute_input":"2023-03-05T13:49:15.093258Z","iopub.status.idle":"2023-03-05T13:49:15.328519Z","shell.execute_reply.started":"2023-03-05T13:49:15.093211Z","shell.execute_reply":"2023-03-05T13:49:15.327249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.query('sign==\"after\"').head()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:49:44.267989Z","iopub.execute_input":"2023-03-05T13:49:44.268341Z","iopub.status.idle":"2023-03-05T13:49:44.294651Z","shell.execute_reply.started":"2023-03-05T13:49:44.268306Z","shell.execute_reply":"2023-03-05T13:49:44.293498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"we use the frist one to Visualization","metadata":{}},{"cell_type":"code","source":"after_landmark = pd.read_parquet(\"/kaggle/input/asl-signs/train_landmark_files/62590/1044729798.parquet\")\nafter_landmark.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:50:42.443549Z","iopub.execute_input":"2023-03-05T13:50:42.444001Z","iopub.status.idle":"2023-03-05T13:50:42.610253Z","shell.execute_reply.started":"2023-03-05T13:50:42.443955Z","shell.execute_reply":"2023-03-05T13:50:42.608876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"unique_frames = after_landmark[\"frame\"].nunique()\nunique_types = after_landmark[\"type\"].nunique()\ntypes_in_video = after_landmark[\"type\"].unique()\nprint(\n    f\"The file has {unique_frames} unique frames and {unique_types} unique types: {types_in_video}\"\n)","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:50:49.712403Z","iopub.execute_input":"2023-03-05T13:50:49.7128Z","iopub.status.idle":"2023-03-05T13:50:49.722334Z","shell.execute_reply.started":"2023-03-05T13:50:49.712763Z","shell.execute_reply":"2023-03-05T13:50:49.721055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"after_landmark.query(\"frame==22 and type=='left_hand'\")","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:51:00.542539Z","iopub.execute_input":"2023-03-05T13:51:00.543013Z","iopub.status.idle":"2023-03-05T13:51:00.567868Z","shell.execute_reply.started":"2023-03-05T13:51:00.54296Z","shell.execute_reply":"2023-03-05T13:51:00.566621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"x\ty\tz  is NaN ,it means `left_hand` does not appear in the frame, we will drop it","metadata":{}},{"cell_type":"code","source":"# define 4 type \ntp_type_holistic=dict(\n    face=mp.solutions.holistic.FACEMESH_CONTOURS,\n    left_hand=mp.solutions.holistic.HAND_CONNECTIONS,\n    right_hand=mp.solutions.holistic.HAND_CONNECTIONS,\n    pose=mp.solutions.holistic.POSE_CONNECTIONS\n)","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:53:38.465619Z","iopub.execute_input":"2023-03-05T13:53:38.466048Z","iopub.status.idle":"2023-03-05T13:53:38.472303Z","shell.execute_reply.started":"2023-03-05T13:53:38.466005Z","shell.execute_reply":"2023-03-05T13:53:38.470842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def show_landmark(filename,size=(512,384,3)):\n    df=pd.read_parquet(filename)\n    unique_frames = df[\"frame\"].nunique()\n    unique_types = df[\"type\"].nunique()\n    types_in_video = df[\"type\"].unique()\n    print(\n        f\"The file has {unique_frames} unique frames and {unique_types} unique types: {types_in_video}\"\n    )\n    df.dropna(inplace=True)\n    unique_frames = df[\"frame\"].nunique()\n    unique_types = df[\"type\"].nunique()\n    types_in_video = df[\"type\"].unique()\n    print(\n        f\"After DropNA file has {unique_frames} unique frames and {unique_types} unique types: {types_in_video}\"\n    )\n    fig, axs = plt.subplots(nrows=unique_frames, ncols=1)\n    fig.set_dpi(700)\n    ind=0\n    for frame in  df[\"frame\"].unique():\n        image=np.zeros(size,np.uint8)\n        for tp in types_in_video:\n            tp_df=df.query(\"frame==@frame and type==@tp\")\n            if tp_df.shape[0]==0:\n                continue\n            ll=[]\n            for row in tp_df.itertuples():\n                ll.append(landmark_pb2.NormalizedLandmark(\n                    x=row.x,\n                    y=row.y,\n                    z=row.z,\n                    visibility=1.0\n                ))\n            landmark_subset = landmark_pb2.NormalizedLandmarkList(\n                  landmark =  ll\n            )\n            mp.solutions.drawing_utils.draw_landmarks(image, landmark_subset, tp_type_holistic[tp])\n        axs[ind].imshow(image)\n        axs[ind].axis('off')\n        ind+=1\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:53:39.159494Z","iopub.execute_input":"2023-03-05T13:53:39.160212Z","iopub.status.idle":"2023-03-05T13:53:39.172282Z","shell.execute_reply.started":"2023-03-05T13:53:39.160165Z","shell.execute_reply":"2023-03-05T13:53:39.171168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_landmark(\"/kaggle/input/asl-signs/train_landmark_files/62590/1044729798.parquet\")","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:53:40.264566Z","iopub.execute_input":"2023-03-05T13:53:40.265337Z","iopub.status.idle":"2023-03-05T13:53:41.01002Z","shell.execute_reply.started":"2023-03-05T13:53:40.265281Z","shell.execute_reply":"2023-03-05T13:53:41.008707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"compare again","metadata":{}},{"cell_type":"code","source":"display_video('/kaggle/working/after_save1.mp4')","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:54:35.891036Z","iopub.execute_input":"2023-03-05T13:54:35.891438Z","iopub.status.idle":"2023-03-05T13:54:35.904216Z","shell.execute_reply.started":"2023-03-05T13:54:35.891403Z","shell.execute_reply":"2023-03-05T13:54:35.902814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"TV ","metadata":{}},{"cell_type":"code","source":"df.query('sign==\"TV\"').head()","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:57:21.876145Z","iopub.execute_input":"2023-03-05T13:57:21.876518Z","iopub.status.idle":"2023-03-05T13:57:21.894558Z","shell.execute_reply.started":"2023-03-05T13:57:21.876482Z","shell.execute_reply":"2023-03-05T13:57:21.893587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_landmark(\"/kaggle/input/asl-signs/train_landmark_files/22343/1003347075.parquet\")","metadata":{"execution":{"iopub.status.busy":"2023-03-05T13:58:25.173911Z","iopub.execute_input":"2023-03-05T13:58:25.174371Z","iopub.status.idle":"2023-03-05T13:58:31.778891Z","shell.execute_reply.started":"2023-03-05T13:58:25.174327Z","shell.execute_reply":"2023-03-05T13:58:31.777585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the quality of data is not as good as we thought,  there are many work we need to do","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}