{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"","metadata":{"id":"xjO2IH-Ko7Hk"}},{"cell_type":"markdown","source":"![](\"https://blogger.googleusercontent.com/img/a/AVvXsEjyOWBxS8yzs3QG7ZnhtrQLDB_UBJPKWYzSa_uPSObkLcrcY9lXjJUopdE4tIve0XA0Wpygb_qpgS-43S5t3EodrpEvbr4JBBLJRDJLirTfxHZahJPad2OZxeY4aZXw3jz3CD5JMqHsHioITsq0oaBj72pzcjTu2YMQIQuTMqqsTq_jJYp2IizyLbw\")
\n\n
\n","metadata":{"id":"l9nL5T1zoKCs"}},{"cell_type":"markdown","source":"# **Google - Isolated Sign Language Recognition competition Step by Step**\n\nThis project is initiated by a Kaggle competition for American sign language with the goal of assisting deaf children's families via Google. Google provides the dataset, which they should modify and use to create a suitable game. In this preliminary project for this competition, our team attempted to train the ASL alphabet using the Mnist dataset and display the results. The accuracy for the alphabet is 99 percent. The model was applied to the camera by our team. A video demonstration displays the outcome of this section of code. The outcome of this demonstration is satisfactory, but the code needs updating and additional steps.\n\n\n1. Start by familiarizing yourself with the problem statement and the dataset provided by the competition. Read the competition overview and the data description carefully.\n\n2. Preprocessing the data: we chose to preprocess the dataset provided by Google. This may involve resizing the images, normalizing pixel values, and splitting the dataset into training, validation, and testing sets.\n\n3. Preparing the model: choose an appropriate DNN architecture for the task of sign language recognition. This may involve selecting a pre-trained model or designing a custom architecture. You will also need to specify the hyperparameters of the model, such as learning rate, batch size, and number of epochs.\n\n4. Training the model: train the DNN model on the training data using a suitable optimizer and loss function. Also monitor the performance of the model on the validation set and adjust the hyperparameters as necessary.\n\n5. Evaluating the model: Once the model is trained, evaluate its performance on the test set. This will give an estimate of the accuracy of the model on new, unseen data.\n\n6. Split the dataset into training, validation, and test sets. Use the training set to train the model, the validation set to tune hyperparameters and prevent overfitting, and the test set to evaluate the performance of the model. Validate the machine learning model using the validation set. Monitor metrics such as accuracy, precision, and recall to ensure that the model is performing well and not overfitting.\n\n7. Fine-tuning the model: Depending on the results of the evaluation, may need to fine-tune the model by adjusting the architecture, hyperparameters, or training procedure. This may involve repeating steps 2-4 multiple times until the desired level of accuracy is achieved. \n\n\nMy team are also using Mediapipe, which is a framework developed by Google that provides a platform for building real-time multimedia processing pipelines offering a comprehensive set of tools for developing computer vision and machine learning applications, including algorithms for detecting, tracking, and recognizing hand gestures, facial expressions, and human poses. Also, it provides a range of built-in tools for visualizing and debugging the output of processing pipelines, making it easier to develop and fine-tune algorithms for specific use cases.\n","metadata":{"id":"zMrnhkxHaFOu"}},{"cell_type":"markdown","source":"# **The accuracy of sign language recognition models applied to real-time camera data may not be satisfactory:**\n\n#Our team experienced problems producing camera output in real-time. There may be one or more of the following reasons why it may be more convenient to have a flawless output:\n\nSign language recognition is a challenging task that has important implications for communication and accessibility. Real-time sign language recognition using cameras has the potential to improve the lives of individuals who are deaf or hard of hearing by enabling them to communicate more easily with others. However, achieving high accuracy with sign language recognition models applied to real-time camera data is a complex task that involves several challenges.\n\nOne of the main challenges is the variability in lighting conditions. Sign language recognition models are often trained on datasets that have consistent lighting conditions, but real-world conditions can vary widely. This can lead to problems when applying the model to new data captured by a camera. Different lighting conditions can affect the contrast, color, and texture of the video data, which can make it difficult for the model to distinguish between different signs.\n\nTo address the challenge of lighting variability, several approaches have been proposed. One approach is to collect training data that includes a wide range of lighting conditions, so that the model can learn to recognize signs under different lighting conditions. Another approach is to use techniques such as histogram equalization or adaptive thresholding to preprocess the video data and enhance the contrast and brightness of the images.\n\nAnother challenge in achieving high accuracy with sign language recognition models applied to real-time camera data is the variability in signers and sign styles. Sign language can vary significantly depending on the individual signer and the style of sign language being used. Models trained on one group of signers or one style of sign language may not generalize well to other groups or styles. For example, American Sign Language (ASL) has different dialects and regional variations, which can make it difficult for a model trained on one dialect to recognize signs from another dialect.\n\nTo address the challenge of signer and style variability, several approaches have been proposed. One approach is to collect training data from a diverse set of signers and styles, so that the model can learn to recognize signs across different individuals and styles. Another approach is to use techniques such as data augmentation or transfer learning to increase the diversity of the training data and improve the model's ability to generalize to new signers and styles.\n\nA third challenge in achieving high accuracy with sign language recognition models applied to real-time camera data is the limited training data. In many cases, the amount of training data available for sign language recognition models is limited. This can make it difficult for the model to learn to recognize a wide range of signs and variations. Additionally, collecting high-quality sign language data can be time-consuming and expensive.\n\nTo address the challenge of limited training data, several approaches have been proposed. One approach is to use techniques such as data augmentation or synthetic data generation to increase the size and diversity of the training data. Another approach is to use transfer learning, where a model trained on a related task can be fine-tuned on a smaller dataset of sign language data to improve its accuracy.\n\nA fourth challenge in achieving high accuracy with sign language recognition models applied to real-time camera data is the limited computing resources. Real-time sign language recognition requires the model to process video data in real-time, which can be computationally intensive. Limited computing resources can lead to slower performance and lower accuracy.\n\nTo address the challenge of limited computing resources, several approaches have been proposed. One approach is to optimize the model architecture and hyperparameters to reduce the computational requirements without sacrificing accuracy. Another approach is to use hardware accelerators such as GPUs or FPGAs to speed up the computation.\n\nIn conclusion, achieving high accuracy with sign language recognition models applied to real-time camera data is a complex task that involves several challenges. These challenges include lighting variability, signer and style variability, limited training data, and limited computing resources. Addressing these challenges.","metadata":{"id":"A03p4kb9aFVL"}},{"cell_type":"markdown","source":"# **ASL Recognition with DNN**\nCommunication sustains us. Deaf and mute people struggle to communicate with hearing people. Differently abled people communicate in many ways, and one of the most common approaches is sign language. To connect with hearing people, deaf and mute people need an app that recognizes sign language. This project uses sign language to bridge hearing and deaf/mute communication. ASL is a complete, natural language with a different grammar than English. ASL uses hands and face movements.\n","metadata":{"id":"UsH8SZ9bx6ZH"}},{"cell_type":"markdown","source":"# **Importing Needed Packages**","metadata":{"id":"TL5qtJQYx6ZJ"}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","id":"VC5uzVuAx6ZJ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import keras\nfrom keras.models import Sequential\nfrom keras.layers import Dense,Flatten,Conv2D,MaxPool2D,Dropout\nimport matplotlib.pyplot as Plot\nimport seaborn as sns\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator","metadata":{"id":"-4hSO6Ejx6ZK"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"from google.colab import drive\ndrive.mount('/content/drive')\"\"\"","metadata":{"id":"DnHqACdP2ckx","outputId":"4cee21bc-70f7-45c6-df91-2832d5372f5f"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{"id":"eQV7vM6PDiPl"}},{"cell_type":"markdown","source":"# **Loading and Preprocessing**\nThe format of the dataset is comparable to the regular MNIST dataset. It contains training and test instances representing labels (0-25), each corresponding to a letter from A to Z. The dataset does not contain examples of the label 9-J and label 25-Z gestures. Each case's format includes a label and a 28x28 pixel image with grayscale values between 0 and 255. This is expressed as the label pixel1, pixel2,..., pixel784 in a header row.","metadata":{"id":"u9RVweY4x6ZK"}},{"cell_type":"code","source":"Train_Data=pd.read_csv('/Users/milindpatil/Downloads/sign_mnist_train.csv')\nTest_Data=pd.read_csv('/Users/milindpatil/Downloads/sign_mnist_test.csv')","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","id":"OodSperDx6ZK"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_Data.describe()","metadata":{"id":"ctpWZz-Fx6ZL","outputId":"06e1787b-e7b7-4cb6-a03d-535b2f681525"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_Data.head(10)","metadata":{"id":"bs7y2fthx6ZL","outputId":"90cf6c54-63b5-4066-892c-1aa76899397c"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_Lab=Train_Data['label']\nTrainset=Train_Data.drop(['label'],axis=1)\n","metadata":{"id":"kVCDb0xsx6ZL"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The data frame is turned into a NumPy array for DNN training. The array is converted from 1-D to 3-D, which is required for the DNN's first layer. A comparable transformation is performed on the test data frame.","metadata":{"id":"KIlukPt9x6ZM"}},{"cell_type":"code","source":"X_Train = Trainset.values\nX_Train = Trainset.values.reshape(-1,28,28,1)\nprint(X_Train.shape)","metadata":{"id":"Ih2UUZZ6x6ZM","outputId":"a425639e-a28d-4d3a-e37e-088362e4d562"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Test_Lab=Test_Data['label']\nX_Test=Test_Data.drop(['label'],axis=1)\nprint(X_Test.shape)\nX_Test.head(10)","metadata":{"id":"kFWO_NWMx6ZM","outputId":"8ae06e14-7382-4488-c270-0227bcdaf2c9"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#**Converting the integer labels to binary form**\nEach image within the label dataframe is assigned a numeric value ranging from 1 to 24. As the classifier must deal with multiple labels, the DNN output layer contains 24 nodes. Each integer is represented by a 24-by-24 binary array with the corresponding label set to 1 and the other labels set to 0. This procedure employs the LabelBinarizer package from the sklearn—preprocessing library.","metadata":{"id":"N4_BBHnnx6ZM"}},{"cell_type":"code","source":"from sklearn.preprocessing import LabelBinarizer\nLabel=LabelBinarizer()\nY_Train=Label.fit_transform(Train_Lab)\nY_Test=Label.fit_transform(Test_Lab)","metadata":{"id":"fWi4mS_Lx6ZM"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Y_Train","metadata":{"id":"gJCKKNASx6ZM","outputId":"ae3ad1b1-ba32-4a85-f7ab-e2b85d70d6c4"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_Test=X_Test.values.reshape(-1,28,28,1)","metadata":{"id":"D9BuWRsHx6ZN"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(X_Train.shape,Y_Train.shape,X_Test.shape,Y_Test.shape)","metadata":{"id":"TvdD6TjBx6ZN","outputId":"902b29a5-8275-4172-f074-f134db6b6aca"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#**Augmentation of the dataset**\n\nThe ImageDataGenerator package in keras.preprocessing.image permits the introduction of various distortions into image datasets by randomly rotating, zooming in/out, and scaling the height or width of every pixel. In addition, the dataset is normalized with the rescale parameter, which divides each pixel by 255 to ensure that pixel values range from 0 to 1.","metadata":{"id":"zIM6vTa3x6ZN"}},{"cell_type":"code","source":"train_datagen = ImageDataGenerator(rescale = 1./255,\n rotation_range = 0,\n height_shift_range=0.2,\n width_shift_range=0.2,\n shear_range=0,\n zoom_range=0.2,\n horizontal_flip=True,\n fill_mode='nearest')\n\nX_Test=X_Test/255","metadata":{"id":"dSyErTQqx6ZN"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Dataset Visualization**","metadata":{"id":"PInQhO-Ix6ZN"}},{"cell_type":"markdown","source":"**Preview of the images in the training dataset**\nThe original images in the dataset are in grayscale format. To display those as it is, use cmap='gray' while displaying the image.For example: axes[0, 0].imshow(X_Train[0].reshape(28,28) cmap='gray')","metadata":{"id":"i1v2b3jOx6ZN"}},{"cell_type":"code","source":"Img, axes = Plot.subplots(2, 3)\nImg.suptitle('Sample Images with the labels\\n\\n')\naxes[0, 0].imshow(X_Train[0].reshape(28,28))\naxes[0, 0].set_title('Label:18, Alphabet: R')\naxes[0, 1].imshow(X_Train[1].reshape(28,28))\naxes[0, 1].set_title('Label:7, Alphabet: G')\naxes[0, 2].imshow(X_Train[2].reshape(28,28))\naxes[0, 2].set_title('Label:3, Alphabet: C')\naxes[1, 0].imshow(X_Train[4].reshape(28,28))\naxes[1, 0].set_title('Label:19, Alphabet: T')\naxes[1, 1].imshow(X_Train[5].reshape(28,28))\naxes[1, 1].set_title('Label:16, Alphabet: P')\naxes[1, 2].imshow(X_Train[6].reshape(28,28))\naxes[1, 2].set_title('Label:1, Alphabet: A')\nPlot.subplots_adjust(wspace=0.5)\nPlot.show()","metadata":{"id":"BH7UYnVQhVu-","outputId":"be587d7a-da23-4911-972d-cd6082ed0655"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Plot the labels**","metadata":{"id":"ioGRdveKx6ZO"}},{"cell_type":"code","source":"import matplotlib.pyplot as Plot\nimport pandas as pd\n\nTrain_Lab_Data = pd.DataFrame(Train_Lab, columns=['label'])\nax = Train_Lab_Data['label'].value_counts().plot(kind='bar')\nax.set_title('Label Frequency')\nPlot.show()","metadata":{"id":"VJ2ZI1HrhVu_","outputId":"a9cfb6a5-8a54-4ee3-8d4a-7234e610774a"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Model Architecture**\n\n","metadata":{"id":"GIOgr8VTx6ZO"}},{"cell_type":"code","source":"from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense, Dropout\nfrom tensorflow.keras.models import Model\n\ninputs = Input(shape=(28, 28, 1))\nx = Conv2D(128, kernel_size=(5,5), strides=1, padding='same', activation='relu')(inputs)\nx = MaxPooling2D(pool_size=(3,3), strides=2, padding='same')(x)\nx = Conv2D(64, kernel_size=(2,2), strides=1, padding='same', activation='relu')(x)\nx = MaxPooling2D(pool_size=(2,2), strides=2, padding='same')(x)\nx = Conv2D(32, kernel_size=(2,2), strides=1, padding='same', activation='relu')(x)\nx = MaxPooling2D(pool_size=(2,2), strides=2, padding='same')(x)\nx = Flatten()(x)\nx = Dense(units=512, activation='relu')(x)\nx = Dropout(rate=0.25)(x)\noutputs = Dense(units=24, activation='softmax')(x)\n\nModel = Model(inputs=inputs, outputs=outputs)\nModel.summary()\n","metadata":{"id":"yXL6npMhhVu_","outputId":"d43bb626-8bd2-4ed7-ebc8-07c1dfb16001"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nModel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['Accuracy'])","metadata":{"id":"jPJudQxBhVu_"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#**Training the model**","metadata":{"id":"DQDLhJOFx6ZP"}},{"cell_type":"code","source":"Model.fit(train_datagen.flow(X_Train,Y_Train,batch_size=200),\n epochs = 40,\n validation_data=(X_Test,Y_Test),\n shuffle=1\n )","metadata":{"id":"sriKtXjhx6ZP","outputId":"35c2948c-1922-47fe-b835-f3c201a026f5"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Applying the model and Measuring the Accuracy**","metadata":{"id":"uxzn8t5Cx6ZP"}},{"cell_type":"code","source":"(ls,acc)=Model.evaluate(x=X_Test,y=Y_Test)","metadata":{"id":"OlAlWweix6ZP","outputId":"9458229b-1436-460e-eb0b-fb9be2acb460"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('MODEL ACCURACY = {}%'.format(acc*100))","metadata":{"id":"H4csJ5z3x6ZP","scrolled":true,"outputId":"70ce640a-92ec-452e-c4af-9aaf08542128"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\nimport seaborn as sns\n\n# Get predictions for test set\nY_Pred = Model.predict(X_Test)\n\n# Convert predictions to labels\nY_Pred = np.argmax(Y_Pred, axis=1)\nY_True = np.argmax(Y_Test, axis=1)\n\n# Compute confusion matrix\nConf_Mat = confusion_matrix(Y_True, Y_Pred)\n\n# Define class labels\nClasses = np.arange(24)\n\n# Plot confusion matrix\nfig, ax = Plot.subplots(figsize=(20, 20))\nim = ax.imshow(Conf_Mat, cmap='Greens')\n\n# Add colorbar\ncbar = ax.figure.colorbar(im, ax=ax)\n\n# Set labels, ticks, and grid\nax.set_xticks(np.arange(len(Classes)))\nax.set_yticks(np.arange(len(Classes)))\nax.set_xticklabels(Classes)\nax.set_yticklabels(Classes)\nax.xaxis.tick_top()\nax.xaxis.set_label_position('top')\nax.set_xlabel('Predicted Label\\n')\nax.set_ylabel('True Label\\n')\nax.grid(False)\n\n# Add text annotations\nThresh = Conf_Mat.max() / 2.\nfor i in range(len(Classes)):\n for j in range(len(Classes)):\n ax.text(j, i, format(Conf_Mat[i, j], 'd'),\n ha=\"center\", va=\"center\",\n color=\"white\" if Conf_Mat[i, j] > Thresh else \"black\")\n\n# Show plot\nPlot.tight_layout()\nPlot.show()","metadata":{"id":"LqqqwYQchVvA","outputId":"bd895264-8e5d-4d86-a53d-488ab2bbf928"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\nimport seaborn as sns\n\n# Get predictions for test set\nY_Pred = Model.predict(X_Test)\n\n# Convert predictions to labels\nY_Pred = np.argmax(Y_Pred, axis=1)\nY_True = np.argmax(Y_Test, axis=1)\n\n# Compute confusion matrix\nConf_Mat = confusion_matrix(Y_True, Y_Pred)\n\n# Compute percentage of each label in the confusion matrix\nTotal_True_Labels = np.sum(Conf_Mat, axis=1)\nConf_Mat_Percentage = Conf_Mat / Total_True_Labels[:, np.newaxis]\n\n# Define class labels\nClasses = np.arange(24)\n\n# Plot confusion matrix\nfig, ax = Plot.subplots(figsize=(20, 20))\nim = ax.imshow(Conf_Mat_Percentage, cmap='Purples')\n\n# Add colorbar\ncbar = ax.figure.colorbar(im, ax=ax)\n\n# Set labels, ticks, and grid\nax.set_xticks(np.arange(len(Classes)))\nax.set_yticks(np.arange(len(Classes)))\nax.set_xticklabels(Classes)\nax.set_yticklabels(Classes)\nax.xaxis.tick_top()\nax.xaxis.set_label_position('top')\nax.set_xlabel('Predicted Label\\n')\nax.set_ylabel('True Label\\n')\nax.grid(False)\n\n# Add text annotations\nfor i in range(len(Classes)):\n for j in range(len(Classes)):\n ax.text(j, i, format(Conf_Mat_Percentage[i, j], '.2%'),\n ha=\"center\", va=\"center\",\n color=\"white\" if Conf_Mat_Percentage[i, j] > 0.5 else \"black\")\n\n# Show plot\nPlot.tight_layout()\nPlot.show()\n","metadata":{"id":"-JslVSEThVvA","outputId":"76bff91c-64de-44fa-dcde-85c66ca862ee"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Model Prediction Using the Camera to Show the Output**","metadata":{"id":"GtjBpnp3HpH0"}},{"cell_type":"code","source":"# Capture the webcam input\nimport cv2\ncap = cv2.VideoCapture(0)\n\nwhile True:\n ret, frame = cap.read()\n\n # Preprocess the input\n input_size = (28, 28)\n preprocessed_frame = cv2.resize(frame, input_size)\n #preprocessed_frame = cv2.resize(frame, input_size, interpolation=cv2.INTER_LINEAR)\n \n preprocessed_frame = cv2.cvtColor(preprocessed_frame, cv2.COLOR_BGR2GRAY)\n preprocessed_frame = np.reshape(preprocessed_frame, (1, *input_size, 1))\n preprocessed_frame = preprocessed_frame / 255.0\n\n # Apply the model to the input\n prediction = Model.predict(preprocessed_frame)\n\n # Postprocess the output\n class_index = np.argmax(prediction)\n class_label = chr(ord('A') + class_index)\n\n # Display the results\n cv2.putText(frame, class_label, (50, 50), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)\n cv2.imshow('Sign Language Recognition', frame)\n\n # Exit on key press\n if cv2.waitKey(1) & 0xFF == ord('q'):\n break","metadata":{"id":"PyjrAlvNni3z"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cap.release()\ncv2.destroyAllWindows()","metadata":{"id":"vRHWWBsFni9W"},"execution_count":null,"outputs":[]}]}