{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Project Title : Sentiment Analysis : Predicting sentiment of COVID-19 tweets","metadata":{"id":"tOGC-qoyhJeX"}},{"cell_type":"markdown","source":"## Problem Description \n\n### This challenge asks you to build a classification model to predict the sentiment of COVID-19 tweets.The tweets have been pulled from Twitter and manual tagging has been done then.\n\n### The names and usernames have been given codes to avoid any privacy concerns.\n\n### You are given the following information:\n1. Location\n2. Tweet At\n3. Original Tweet\n4. Label","metadata":{"id":"Y06xIdG26kRF"}},{"cell_type":"markdown","source":"","metadata":{"id":"HnVygsLkmEnc"}},{"cell_type":"markdown","source":"COVID-19 originally known as Corona VIrus Disease of 2019, has been declared as a pandemic by World Health Organization (WHO) on 11th March 2020. Unprecedented pressures have mounted on each country to make compelling requisites for controlling the population by assessing the cases and properly utilizing available resources. The rapid number of exponential cases globally has become the apprehension of panic, fear and anxiety among people. The mental and physical health of the global population is found to be directly proportional to this pandemic disease. It is the need of the hour to implement different measures to safeguard the countries by demystifying the pertinent facts and information.","metadata":{"id":"i8e0fQFtnXOn"}},{"cell_type":"markdown","source":"# Workflow","metadata":{"id":"Qxo3tpqkm8rt"}},{"cell_type":"markdown","source":"","metadata":{"id":"oO_hpKGnnCq_"}},{"cell_type":"markdown","source":"# **1.Lets Invite Some Libraries.**","metadata":{"id":"k1oDG6-aVZry"}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom numpy import percentile\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport matplotlib\nimport re # for regular expressions\nimport pandas as pd \npd.set_option(\"display.max_colwidth\", 200) \nimport string\nimport nltk # for text manipulation\nfrom nltk.stem.porter import *\nfrom sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\nfrom tqdm import tqdm\n#from gensim.models.doc2vec import LabeledSentence\nimport gensim\nfrom sklearn.linear_model import LogisticRegression\nfrom scipy import stats \nfrom sklearn import metrics \nfrom sklearn.metrics import mean_squared_error,mean_absolute_error, make_scorer,classification_report,confusion_matrix,accuracy_score,roc_auc_score,roc_curve\nfrom sklearn.model_selection import train_test_split,cross_val_score,KFold\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import f1_score\nfrom sklearn.naive_bayes import BernoulliNB\nfrom sklearn import svm\nfrom sklearn.ensemble import RandomForestClassifier\nfrom xgboost import XGBClassifier\nimport xgboost as xgb\nimport warnings \nwarnings.filterwarnings(\"ignore\")\n\n%matplotlib inline","metadata":{"id":"wP3hkPGC6Q5P","execution":{"iopub.status.busy":"2022-09-19T06:10:09.30216Z","iopub.execute_input":"2022-09-19T06:10:09.302796Z","iopub.status.idle":"2022-09-19T06:10:09.566222Z","shell.execute_reply.started":"2022-09-19T06:10:09.302733Z","shell.execute_reply":"2022-09-19T06:10:09.565054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv('../input/covid19-tweets-data/Corona_NLP_train.csv',encoding = 'latin-1')","metadata":{"id":"Dp-fSiTQ5vHc","execution":{"iopub.status.busy":"2022-09-19T06:19:48.248726Z","iopub.execute_input":"2022-09-19T06:19:48.249164Z","iopub.status.idle":"2022-09-19T06:19:48.417645Z","shell.execute_reply.started":"2022-09-19T06:19:48.249128Z","shell.execute_reply":"2022-09-19T06:19:48.4167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **2. Reviewing Our Dataset**","metadata":{"id":"kpbpiYirVmdE"}},{"cell_type":"code","source":"df.head()","metadata":{"id":"eFhWhGl16Kf3","outputId":"06e352ed-0eb9-4a7d-e8af-f403b4c11b5c","execution":{"iopub.status.busy":"2022-09-19T06:19:48.868547Z","iopub.execute_input":"2022-09-19T06:19:48.869272Z","iopub.status.idle":"2022-09-19T06:19:48.883211Z","shell.execute_reply.started":"2022-09-19T06:19:48.869212Z","shell.execute_reply":"2022-09-19T06:19:48.881902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# There are null values in location column\ndf.info()","metadata":{"id":"o5AVNH4dRPoQ","outputId":"5dd78d21-194d-4621-baf7-107f141db3d8","execution":{"iopub.status.busy":"2022-09-19T06:19:49.445497Z","iopub.execute_input":"2022-09-19T06:19:49.445927Z","iopub.status.idle":"2022-09-19T06:19:49.487608Z","shell.execute_reply.started":"2022-09-19T06:19:49.445887Z","shell.execute_reply":"2022-09-19T06:19:49.486776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# There 41157 rows and 6 columns in our database\ndf.shape","metadata":{"id":"N9txhbay1v5D","outputId":"8742db0f-e639-43f9-bf25-39e328929163","execution":{"iopub.status.busy":"2022-09-19T06:19:49.811823Z","iopub.execute_input":"2022-09-19T06:19:49.812579Z","iopub.status.idle":"2022-09-19T06:19:49.818895Z","shell.execute_reply.started":"2022-09-19T06:19:49.812539Z","shell.execute_reply":"2022-09-19T06:19:49.817765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Names Of columns in our dataset\ndf.columns","metadata":{"id":"D2u-sX3w11RG","outputId":"6dd6dacb-9c83-4818-c8dd-a14bed126d46","execution":{"iopub.status.busy":"2022-09-19T06:19:50.163357Z","iopub.execute_input":"2022-09-19T06:19:50.164068Z","iopub.status.idle":"2022-09-19T06:19:50.170607Z","shell.execute_reply.started":"2022-09-19T06:19:50.164028Z","shell.execute_reply":"2022-09-19T06:19:50.169522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# All tweets came only from March and April month of 2020.\ndf.TweetAt.value_counts()","metadata":{"id":"W95WJfHz1_z1","outputId":"456cf301-1dea-4d8f-8ac4-aca09cff02ae","execution":{"iopub.status.busy":"2022-09-19T06:19:50.553272Z","iopub.execute_input":"2022-09-19T06:19:50.553988Z","iopub.status.idle":"2022-09-19T06:19:50.56434Z","shell.execute_reply.started":"2022-09-19T06:19:50.55395Z","shell.execute_reply":"2022-09-19T06:19:50.563287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# There are 12220 unique locations from where the tweets came.\ndf['Location'].value_counts()","metadata":{"id":"wIedPb7W2FRA","outputId":"9d94dde6-919b-48f3-f20f-1cf5001c0c21","execution":{"iopub.status.busy":"2022-09-19T06:19:50.933937Z","iopub.execute_input":"2022-09-19T06:19:50.935157Z","iopub.status.idle":"2022-09-19T06:19:50.954164Z","shell.execute_reply.started":"2022-09-19T06:19:50.935087Z","shell.execute_reply":"2022-09-19T06:19:50.952989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# There are 5 unique sentiment types in our dataset\ndf['Sentiment'].value_counts()","metadata":{"id":"6d8oFpWx2KYO","outputId":"eb629a0e-aaf7-481f-c891-47ba7b7e2ec1","execution":{"iopub.status.busy":"2022-09-19T06:19:51.202002Z","iopub.execute_input":"2022-09-19T06:19:51.202672Z","iopub.status.idle":"2022-09-19T06:19:51.212911Z","shell.execute_reply.started":"2022-09-19T06:19:51.20263Z","shell.execute_reply":"2022-09-19T06:19:51.211896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Going deeper into tweet and sentiment.","metadata":{"id":"2bhPVfvYyYC6"}},{"cell_type":"code","source":"df[df['Sentiment']=='Extremely Negative']","metadata":{"id":"oJc1PQKR2Smw","outputId":"be945e8d-c139-47a5-e128-b055574ff937","execution":{"iopub.status.busy":"2022-09-19T06:19:51.872976Z","iopub.execute_input":"2022-09-19T06:19:51.873406Z","iopub.status.idle":"2022-09-19T06:19:51.895612Z","shell.execute_reply.started":"2022-09-19T06:19:51.873365Z","shell.execute_reply":"2022-09-19T06:19:51.894464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df['Sentiment']=='Extremely Negative'].loc[20,'OriginalTweet']","metadata":{"id":"vquE1Xc12g3A","outputId":"36302b32-007f-44d9-eb0c-c9e852b6bee0","execution":{"iopub.status.busy":"2022-09-19T06:19:52.214206Z","iopub.execute_input":"2022-09-19T06:19:52.214616Z","iopub.status.idle":"2022-09-19T06:19:52.228199Z","shell.execute_reply.started":"2022-09-19T06:19:52.214578Z","shell.execute_reply":"2022-09-19T06:19:52.226915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df['Sentiment']=='Extremely Negative'].loc[41153,'OriginalTweet']","metadata":{"id":"455kCW2g2lgf","outputId":"c57453f4-3694-440c-fc77-743a6072f1ed","execution":{"iopub.status.busy":"2022-09-19T06:19:52.532153Z","iopub.execute_input":"2022-09-19T06:19:52.533376Z","iopub.status.idle":"2022-09-19T06:19:52.547621Z","shell.execute_reply.started":"2022-09-19T06:19:52.533325Z","shell.execute_reply":"2022-09-19T06:19:52.546363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df['Sentiment']=='Negative']","metadata":{"id":"KW5riNQF2uB6","outputId":"8a3fbc2d-1234-449c-dd6f-3e52dfdcdc47","execution":{"iopub.status.busy":"2022-09-19T06:19:52.830386Z","iopub.execute_input":"2022-09-19T06:19:52.831134Z","iopub.status.idle":"2022-09-19T06:19:52.853946Z","shell.execute_reply.started":"2022-09-19T06:19:52.83109Z","shell.execute_reply":"2022-09-19T06:19:52.852646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df['Sentiment']=='Negative'].loc[41149,'OriginalTweet']","metadata":{"id":"j_t-rVPl21dx","outputId":"29707b3e-ae17-4a1f-8816-d3369b1c61f0","execution":{"iopub.status.busy":"2022-09-19T06:19:53.165886Z","iopub.execute_input":"2022-09-19T06:19:53.166911Z","iopub.status.idle":"2022-09-19T06:19:53.183463Z","shell.execute_reply.started":"2022-09-19T06:19:53.166838Z","shell.execute_reply":"2022-09-19T06:19:53.182182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df['Sentiment']=='Neutral']","metadata":{"id":"B_EBsnOB29Yo","outputId":"7bd1adfc-01d5-499b-f3dd-f226b2c121f7","execution":{"iopub.status.busy":"2022-09-19T06:19:53.427812Z","iopub.execute_input":"2022-09-19T06:19:53.428236Z","iopub.status.idle":"2022-09-19T06:19:53.450868Z","shell.execute_reply.started":"2022-09-19T06:19:53.428201Z","shell.execute_reply":"2022-09-19T06:19:53.449681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df['Sentiment']=='Neutral'].loc[17,'OriginalTweet']","metadata":{"id":"nbUvdG1Z3Doq","outputId":"584a6475-c215-4a2b-cb4d-1ffc6a328b38","execution":{"iopub.status.busy":"2022-09-19T06:19:53.752742Z","iopub.execute_input":"2022-09-19T06:19:53.753477Z","iopub.status.idle":"2022-09-19T06:19:53.766784Z","shell.execute_reply.started":"2022-09-19T06:19:53.753428Z","shell.execute_reply":"2022-09-19T06:19:53.765643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df['Sentiment']=='Neutral'].loc[16,'OriginalTweet']","metadata":{"id":"iCRQjSTH3ME4","outputId":"8107afe5-af6d-4a75-9610-94645dae758e","execution":{"iopub.status.busy":"2022-09-19T06:19:54.046601Z","iopub.execute_input":"2022-09-19T06:19:54.047995Z","iopub.status.idle":"2022-09-19T06:19:54.061306Z","shell.execute_reply.started":"2022-09-19T06:19:54.047943Z","shell.execute_reply":"2022-09-19T06:19:54.060069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df['Sentiment']=='Positive']","metadata":{"id":"CoVNI6hH3RqV","outputId":"eecd5140-3454-423f-8a38-636b89ce260d","execution":{"iopub.status.busy":"2022-09-19T06:19:54.370759Z","iopub.execute_input":"2022-09-19T06:19:54.371159Z","iopub.status.idle":"2022-09-19T06:19:54.394492Z","shell.execute_reply.started":"2022-09-19T06:19:54.371126Z","shell.execute_reply":"2022-09-19T06:19:54.393358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df['Sentiment']=='Positive'].loc[1,'OriginalTweet']","metadata":{"id":"TzeTCpR63Xaj","outputId":"1f86543f-e538-466c-d90f-3697eac9560d","execution":{"iopub.status.busy":"2022-09-19T06:19:54.659221Z","iopub.execute_input":"2022-09-19T06:19:54.659614Z","iopub.status.idle":"2022-09-19T06:19:54.676013Z","shell.execute_reply.started":"2022-09-19T06:19:54.65958Z","shell.execute_reply":"2022-09-19T06:19:54.674747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df['Sentiment']=='Positive'].loc[6,'OriginalTweet']","metadata":{"id":"ny8vicF-3fHs","outputId":"bd676aca-bec5-4eed-bcb1-0a97d11ba373","execution":{"iopub.status.busy":"2022-09-19T06:19:54.972823Z","iopub.execute_input":"2022-09-19T06:19:54.973282Z","iopub.status.idle":"2022-09-19T06:19:54.987521Z","shell.execute_reply.started":"2022-09-19T06:19:54.973212Z","shell.execute_reply":"2022-09-19T06:19:54.986269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df['Sentiment']=='Extremely Positive']","metadata":{"id":"OInAsHC-3irU","outputId":"82efb25b-3853-45e2-a0e8-ff0c6575ac1a","execution":{"iopub.status.busy":"2022-09-19T06:19:55.442904Z","iopub.execute_input":"2022-09-19T06:19:55.443906Z","iopub.status.idle":"2022-09-19T06:19:55.464917Z","shell.execute_reply.started":"2022-09-19T06:19:55.443866Z","shell.execute_reply":"2022-09-19T06:19:55.463831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df['Sentiment']=='Extremely Positive'].loc[11,'OriginalTweet']","metadata":{"id":"q43o4pue3oF3","outputId":"8fad939a-9528-410f-a6cd-9862f61fac1d","execution":{"iopub.status.busy":"2022-09-19T06:19:55.691418Z","iopub.execute_input":"2022-09-19T06:19:55.691842Z","iopub.status.idle":"2022-09-19T06:19:55.705722Z","shell.execute_reply.started":"2022-09-19T06:19:55.691808Z","shell.execute_reply":"2022-09-19T06:19:55.704374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df['Sentiment']=='Extremely Positive'].loc[22,'OriginalTweet']","metadata":{"id":"ztPjS6Ap4MxX","outputId":"26fc7b53-6548-4e20-b001-639c4191d375","execution":{"iopub.status.busy":"2022-09-19T06:19:55.910779Z","iopub.execute_input":"2022-09-19T06:19:55.911631Z","iopub.status.idle":"2022-09-19T06:19:55.925327Z","shell.execute_reply.started":"2022-09-19T06:19:55.911587Z","shell.execute_reply":"2022-09-19T06:19:55.924049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.isnull(df[\"Sentiment\"])\n","metadata":{"id":"VkPl7zZy4PfM","outputId":"ecdb6ebe-8590-4c05-f39f-78a4c97a338a","execution":{"iopub.status.busy":"2022-09-19T06:19:56.29496Z","iopub.execute_input":"2022-09-19T06:19:56.295397Z","iopub.status.idle":"2022-09-19T06:19:56.30664Z","shell.execute_reply.started":"2022-09-19T06:19:56.295358Z","shell.execute_reply":"2022-09-19T06:19:56.305365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Looking For Null Values","metadata":{"id":"3CH6_O1OQVbX"}},{"cell_type":"markdown","source":"Only Location column contains 28% null values. But this column is not useful for our sentiment analysis.Hence we will neglect these null values.","metadata":{"id":"EeShg5ECQljd"}},{"cell_type":"code","source":"missing_values = pd.DataFrame()\nmissing_values['column'] = df.columns\n\nmissing_values['percent'] = [round(100* df[col].isnull().sum() / len(df), 2) for col in df.columns]\nmissing_values = missing_values.sort_values('percent')\nmissing_values = missing_values[missing_values['percent']>0]","metadata":{"id":"DF5VoEluQY5g","execution":{"iopub.status.busy":"2022-09-19T06:19:57.200792Z","iopub.execute_input":"2022-09-19T06:19:57.201253Z","iopub.status.idle":"2022-09-19T06:19:57.225892Z","shell.execute_reply.started":"2022-09-19T06:19:57.201214Z","shell.execute_reply":"2022-09-19T06:19:57.224719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\nsns.set(style='whitegrid', color_codes=True)\nsplot=sns.barplot(x='column', y='percent', data=missing_values)\nfor p in splot.patches:\n splot.annotate(format(p.get_height(), '.2f'), (p.get_x() + p.get_width() / 2., p.get_height()), ha = 'center',\n va = 'center', xytext = (0, 9), textcoords = 'offset points')\nplt.xlabel(\"Column_Name\", size=14, weight=\"bold\")\nplt.ylabel(\"Percentage\", size=14, weight=\"bold\")\nplt.title(\"Percentage of missing values in column\",fontweight=\"bold\",size=17)\nplt.show()\n","metadata":{"id":"r5XIKQELQgds","outputId":"fd83fc9c-a73d-4c99-a5da-df874443b17f","execution":{"iopub.status.busy":"2022-09-19T06:19:57.51113Z","iopub.execute_input":"2022-09-19T06:19:57.511893Z","iopub.status.idle":"2022-09-19T06:19:57.770494Z","shell.execute_reply.started":"2022-09-19T06:19:57.511836Z","shell.execute_reply":"2022-09-19T06:19:57.769341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Heat Map for missing values","metadata":{"id":"-dcnAE2NRPQ7"}},{"cell_type":"code","source":"plt.figure(figsize=(17, 5))\nsns.heatmap(df.isnull(), cbar=True, yticklabels=False)\nplt.xlabel(\"Column_Name\", size=14, weight=\"bold\")\nplt.title(\"Places of missing values in column\",fontweight=\"bold\",size=17)\nplt.show()","metadata":{"id":"xUNvfEtTRS9r","outputId":"58a434c7-3c41-4455-fdeb-652231deeae9","execution":{"iopub.status.busy":"2022-09-19T06:19:58.466093Z","iopub.execute_input":"2022-09-19T06:19:58.466535Z","iopub.status.idle":"2022-09-19T06:19:58.984109Z","shell.execute_reply.started":"2022-09-19T06:19:58.466496Z","shell.execute_reply":"2022-09-19T06:19:58.98292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Unique Values In Each Feature Coulmn","metadata":{"id":"AtHRefCSRejx"}},{"cell_type":"code","source":"unique_df = pd.DataFrame()\nunique_df['Features'] = df.columns\nunique=[]\nfor i in df.columns:\n unique.append(df[i].nunique())\nunique_df['Uniques'] = unique\n\nf, ax = plt.subplots(1,1, figsize=(15,7))\n\nsplot = sns.barplot(x=unique_df['Features'], y=unique_df['Uniques'], alpha=0.8)\nfor p in splot.patches:\n splot.annotate(format(p.get_height(), '.0f'), (p.get_x() + p.get_width() / 2., p.get_height()), ha = 'center',\n va = 'center', xytext = (0, 9), textcoords = 'offset points')\nplt.title('Bar plot for number of unique values in each column',weight='bold', size=15)\nplt.ylabel('#Unique values', size=12, weight='bold')\nplt.xlabel('Features', size=12, weight='bold')\nplt.xticks(rotation=90)\nplt.show()","metadata":{"id":"oGLXSnivRdsH","outputId":"c425de7b-c99f-48c7-967b-a2e3747bf160","execution":{"iopub.status.busy":"2022-09-19T06:19:59.076397Z","iopub.execute_input":"2022-09-19T06:19:59.07705Z","iopub.status.idle":"2022-09-19T06:19:59.423891Z","shell.execute_reply.started":"2022-09-19T06:19:59.07701Z","shell.execute_reply":"2022-09-19T06:19:59.422794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Of Top 15 Locations Of Tweet.","metadata":{"id":"n9gCFW4XTQ5k"}},{"cell_type":"code","source":"loc_analysis = pd.DataFrame(df['Location'].value_counts().sort_values(ascending=False))\nloc_analysis = loc_analysis.rename(columns={'Location':'count'})","metadata":{"id":"ER79KekESOxh","execution":{"iopub.status.busy":"2022-09-19T06:19:59.758179Z","iopub.execute_input":"2022-09-19T06:19:59.758869Z","iopub.status.idle":"2022-09-19T06:19:59.774357Z","shell.execute_reply.started":"2022-09-19T06:19:59.758814Z","shell.execute_reply":"2022-09-19T06:19:59.773268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.graph_objects as go\n","metadata":{"id":"q3aXXgNPTKc3","execution":{"iopub.status.busy":"2022-09-19T06:20:00.047384Z","iopub.execute_input":"2022-09-19T06:20:00.047786Z","iopub.status.idle":"2022-09-19T06:20:00.060799Z","shell.execute_reply.started":"2022-09-19T06:20:00.047751Z","shell.execute_reply":"2022-09-19T06:20:00.059413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = {\n \"values\": loc_analysis['count'][:15],\n \"labels\": loc_analysis.index[:15],\n \"domain\": {\"column\": 0},\n \"name\": \"Location Name\",\n \"hoverinfo\":\"label+percent+name\",\n \"hole\": .4,\n \"type\": \"pie\"\n}\nlayout = go.Layout(title=\"Ratio on Location\", legend=dict(x=0.1, y=1.1, orientation=\"h\"))\n\ndata = [data]\nfig = go.Figure(data = data, layout = layout)\nfig.update_layout(title_x=0.5)\nfig.show()","metadata":{"id":"WudbzTuzSObY","outputId":"df58da13-cb47-4947-f5e9-e27fe1a2761d","execution":{"iopub.status.busy":"2022-09-19T06:20:00.401289Z","iopub.execute_input":"2022-09-19T06:20:00.401695Z","iopub.status.idle":"2022-09-19T06:20:00.723186Z","shell.execute_reply.started":"2022-09-19T06:20:00.401662Z","shell.execute_reply":"2022-09-19T06:20:00.722081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.catplot(\"TweetAt\", data=df, kind=\"count\", height=8)","metadata":{"id":"NaAdAkuk5FZX","outputId":"c257df31-7d74-47e0-d736-ea043e6613b0","execution":{"iopub.status.busy":"2022-09-19T06:20:00.725576Z","iopub.execute_input":"2022-09-19T06:20:00.726197Z","iopub.status.idle":"2022-09-19T06:20:01.429083Z","shell.execute_reply.started":"2022-09-19T06:20:00.726151Z","shell.execute_reply":"2022-09-19T06:20:01.427976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Top 10 locations of tweet","metadata":{"id":"_TcZhTy7TgIO"}},{"cell_type":"code","source":"Top_Location_Of_tweet= df['Location'].value_counts().head(10)\n\nsns.set(rc={'figure.figsize':(12,8)})\nsns.set_style('white')","metadata":{"id":"35ZeE6TSIlnp","execution":{"iopub.status.busy":"2022-09-19T06:20:01.430605Z","iopub.execute_input":"2022-09-19T06:20:01.430957Z","iopub.status.idle":"2022-09-19T06:20:01.444162Z","shell.execute_reply.started":"2022-09-19T06:20:01.430925Z","shell.execute_reply":"2022-09-19T06:20:01.443206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Top_Location_Of_tweet.head(10)","metadata":{"id":"2hPXLQCUJ2cC","outputId":"fc31f852-ac87-4201-f916-f79671bdd784","execution":{"iopub.status.busy":"2022-09-19T06:20:01.741095Z","iopub.execute_input":"2022-09-19T06:20:01.742022Z","iopub.status.idle":"2022-09-19T06:20:01.750449Z","shell.execute_reply.started":"2022-09-19T06:20:01.741976Z","shell.execute_reply":"2022-09-19T06:20:01.749216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Top_Location_Of_tweet_df=pd.DataFrame(Top_Location_Of_tweet)\nTop_Location_Of_tweet_df.reset_index(inplace=True)\nTop_Location_Of_tweet_df.rename(columns={'index':'Location', 'Location':'Location_Count'}, inplace=True)\nTop_Location_Of_tweet_df","metadata":{"id":"yltVOxi9JnkQ","outputId":"4eaf4836-9ec9-452d-8553-562e6caa6aa8","execution":{"iopub.status.busy":"2022-09-19T06:20:02.036167Z","iopub.execute_input":"2022-09-19T06:20:02.03657Z","iopub.status.idle":"2022-09-19T06:20:02.049363Z","shell.execute_reply.started":"2022-09-19T06:20:02.036537Z","shell.execute_reply":"2022-09-19T06:20:02.048266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"viz_1=sns.barplot(x=\"Location\", y=\"Location_Count\", data=Top_Location_Of_tweet_df,\n palette='Blues_d')\nviz_1.set_title('Locations with most of the tweets')\nviz_1.set_ylabel('Count of listings')\nviz_1.set_xlabel('Location Names')\nviz_1.set_xticklabels(viz_1.get_xticklabels(), rotation=45)","metadata":{"id":"FXPxupIBKs0A","outputId":"c82469b4-6bae-4441-e335-fefedc76e9eb","execution":{"iopub.status.busy":"2022-09-19T06:20:02.230555Z","iopub.execute_input":"2022-09-19T06:20:02.231022Z","iopub.status.idle":"2022-09-19T06:20:02.5588Z","shell.execute_reply.started":"2022-09-19T06:20:02.230984Z","shell.execute_reply":"2022-09-19T06:20:02.55761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Sentiments Countplot","metadata":{"id":"lYBS9WamToBk"}},{"cell_type":"code","source":"sns.set(font_scale=1.1)\nsns.catplot(\"Sentiment\", data=df, kind=\"count\", height=8)","metadata":{"id":"JXPL7YB4LMe8","outputId":"b13561d4-5ea0-40df-ce98-cd3082fc88eb","execution":{"iopub.status.busy":"2022-09-19T06:20:02.883282Z","iopub.execute_input":"2022-09-19T06:20:02.88368Z","iopub.status.idle":"2022-09-19T06:20:03.280899Z","shell.execute_reply.started":"2022-09-19T06:20:02.883647Z","shell.execute_reply":"2022-09-19T06:20:03.27966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.TweetAt.value_counts()","metadata":{"id":"YAs6Sj77cFB-","outputId":"fc533350-da68-487c-e95c-3d78d9fcb912","execution":{"iopub.status.busy":"2022-09-19T06:20:03.336691Z","iopub.execute_input":"2022-09-19T06:20:03.337971Z","iopub.status.idle":"2022-09-19T06:20:03.350055Z","shell.execute_reply.started":"2022-09-19T06:20:03.337908Z","shell.execute_reply":"2022-09-19T06:20:03.348827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preprocessing","metadata":{"id":"KmWXbOtC2kk8"}},{"cell_type":"markdown","source":"**A) Removing @user**","metadata":{"id":"CGxa92wN3cyc"}},{"cell_type":"code","source":"# write function for removing @user\ndef remove_pattern(input_txt, pattern):\n r = re.findall(pattern, input_txt)\n for i in r:\n input_txt = re.sub(i,'',input_txt)\n return input_txt","metadata":{"id":"qm3l84ZFzSa8","execution":{"iopub.status.busy":"2022-09-19T06:20:04.199395Z","iopub.execute_input":"2022-09-19T06:20:04.20053Z","iopub.status.idle":"2022-09-19T06:20:04.20657Z","shell.execute_reply.started":"2022-09-19T06:20:04.200482Z","shell.execute_reply":"2022-09-19T06:20:04.205302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create new column with removed @user\ndf['Tweet'] = np.vectorize(remove_pattern)(df['OriginalTweet'], '@[\\w]*')","metadata":{"id":"K90ogxco2_Xg","execution":{"iopub.status.busy":"2022-09-19T06:20:04.52927Z","iopub.execute_input":"2022-09-19T06:20:04.530767Z","iopub.status.idle":"2022-09-19T06:20:05.587361Z","shell.execute_reply.started":"2022-09-19T06:20:04.530708Z","shell.execute_reply":"2022-09-19T06:20:05.586433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(2)","metadata":{"id":"GYQNpBEbISMA","outputId":"49b5a427-54ae-49f3-f57e-e36ec751d8ca","execution":{"iopub.status.busy":"2022-09-19T06:20:05.588837Z","iopub.execute_input":"2022-09-19T06:20:05.589842Z","iopub.status.idle":"2022-09-19T06:20:05.606699Z","shell.execute_reply.started":"2022-09-19T06:20:05.589807Z","shell.execute_reply":"2022-09-19T06:20:05.605561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**REMOVED HTTP AND URLS FROM TWEET**","metadata":{"id":"IKidshkKVDg3"}},{"cell_type":"code","source":"import re\ndf['Tweet'] = df['Tweet'].apply(lambda x: re.split('https:\\/\\/.*', str(x))[0])","metadata":{"id":"pwFIAq1pTsue","execution":{"iopub.status.busy":"2022-09-19T06:20:05.608007Z","iopub.execute_input":"2022-09-19T06:20:05.608629Z","iopub.status.idle":"2022-09-19T06:20:05.698661Z","shell.execute_reply.started":"2022-09-19T06:20:05.608593Z","shell.execute_reply":"2022-09-19T06:20:05.697721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(3)","metadata":{"id":"pnd0eT4JU6Wv","outputId":"9d96b844-fe42-4d98-d86c-9e86cae2c9b7","execution":{"iopub.status.busy":"2022-09-19T06:20:06.941306Z","iopub.execute_input":"2022-09-19T06:20:06.942024Z","iopub.status.idle":"2022-09-19T06:20:06.955634Z","shell.execute_reply.started":"2022-09-19T06:20:06.941978Z","shell.execute_reply":"2022-09-19T06:20:06.954544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**B) Removing Punctuations, Numbers, and Special Characters**","metadata":{"id":"JVIrqhtu3Z8q"}},{"cell_type":"code","source":"# remove special characters, numbers, punctuations\ndf['Tweet'] = df['Tweet'].str.replace('[^a-zA-Z#]+',' ')","metadata":{"id":"0O-bQIpK3ZGH","execution":{"iopub.status.busy":"2022-09-19T06:20:07.837171Z","iopub.execute_input":"2022-09-19T06:20:07.8376Z","iopub.status.idle":"2022-09-19T06:20:08.287531Z","shell.execute_reply.started":"2022-09-19T06:20:07.837567Z","shell.execute_reply":"2022-09-19T06:20:08.2863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(5)","metadata":{"id":"WqVZBuqfRiu6","outputId":"d8d23828-872b-4219-fc16-04fb8bfd0ea0","execution":{"iopub.status.busy":"2022-09-19T06:20:08.289397Z","iopub.execute_input":"2022-09-19T06:20:08.289783Z","iopub.status.idle":"2022-09-19T06:20:08.305454Z","shell.execute_reply.started":"2022-09-19T06:20:08.289745Z","shell.execute_reply":"2022-09-19T06:20:08.304227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**C) Removing Short Words**","metadata":{"id":"ooiAoESF33ST"}},{"cell_type":"code","source":"# remove short words\ndf['Tweet'] = df['Tweet'].apply(lambda x: ' '.join([w for w in x.split() if len(w) > 2]))","metadata":{"id":"sSfiEeOz3OB4","execution":{"iopub.status.busy":"2022-09-19T06:20:08.944072Z","iopub.execute_input":"2022-09-19T06:20:08.944711Z","iopub.status.idle":"2022-09-19T06:20:09.229776Z","shell.execute_reply.started":"2022-09-19T06:20:08.944664Z","shell.execute_reply":"2022-09-19T06:20:09.228931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(2)","metadata":{"id":"5v_IjnXRSLch","outputId":"c78cc133-18f8-498b-cef8-2c85b72d6d34","execution":{"iopub.status.busy":"2022-09-19T06:20:09.34492Z","iopub.execute_input":"2022-09-19T06:20:09.345596Z","iopub.status.idle":"2022-09-19T06:20:09.35922Z","shell.execute_reply.started":"2022-09-19T06:20:09.345549Z","shell.execute_reply":"2022-09-19T06:20:09.35794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"OVty5YQ5rfKd"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**D) Tokenization**","metadata":{"id":"muENaDu24HM6"}},{"cell_type":"code","source":"# create new variable tokenized tweet \ntokenized_tweet = df['Tweet'].apply(lambda x: x.split())","metadata":{"id":"jg1erD884D_9","execution":{"iopub.status.busy":"2022-09-19T06:20:10.980737Z","iopub.execute_input":"2022-09-19T06:20:10.981177Z","iopub.status.idle":"2022-09-19T06:20:11.12261Z","shell.execute_reply.started":"2022-09-19T06:20:10.981142Z","shell.execute_reply":"2022-09-19T06:20:11.1212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(2)","metadata":{"id":"MuFotRJtWQYP","outputId":"d29c843e-59c9-4c13-a26b-d4e617f35151","execution":{"iopub.status.busy":"2022-09-19T06:20:12.002954Z","iopub.execute_input":"2022-09-19T06:20:12.00338Z","iopub.status.idle":"2022-09-19T06:20:12.016582Z","shell.execute_reply.started":"2022-09-19T06:20:12.003338Z","shell.execute_reply":"2022-09-19T06:20:12.015188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**E) Stemming**","metadata":{"id":"8h_UDv0E4S64"}},{"cell_type":"code","source":"from nltk.stem.porter import *\nstemmer = PorterStemmer()\n\n# apply stemmer for tokenized_tweet\ntokenized_tweet = tokenized_tweet.apply(lambda x: [stemmer.stem(i) for i in x])","metadata":{"id":"ygPhahRD4Oq5","execution":{"iopub.status.busy":"2022-09-19T06:20:13.564727Z","iopub.execute_input":"2022-09-19T06:20:13.565141Z","iopub.status.idle":"2022-09-19T06:20:40.94693Z","shell.execute_reply.started":"2022-09-19T06:20:13.565106Z","shell.execute_reply":"2022-09-19T06:20:40.945536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(2)","metadata":{"id":"a3K9_QnjWeYR","outputId":"7509c6c6-cde9-4f17-dac1-22ea76e7c6b6","execution":{"iopub.status.busy":"2022-09-19T06:20:40.948788Z","iopub.execute_input":"2022-09-19T06:20:40.949207Z","iopub.status.idle":"2022-09-19T06:20:40.964873Z","shell.execute_reply.started":"2022-09-19T06:20:40.949165Z","shell.execute_reply":"2022-09-19T06:20:40.963532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# join tokens into one sentence\nfor i in range(len(tokenized_tweet)):\n tokenized_tweet[i] = ' '.join(tokenized_tweet[i])\n# change df['Tweet'] to tokenized_tweet","metadata":{"id":"WAxVp_W040i6","execution":{"iopub.status.busy":"2022-09-19T06:21:08.069788Z","iopub.execute_input":"2022-09-19T06:21:08.07029Z","iopub.status.idle":"2022-09-19T06:21:08.531204Z","shell.execute_reply.started":"2022-09-19T06:21:08.070248Z","shell.execute_reply":"2022-09-19T06:21:08.530015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['Tweet'] = tokenized_tweet\n","metadata":{"id":"V4-fGcF-47Ff","execution":{"iopub.status.busy":"2022-09-19T06:21:10.08083Z","iopub.execute_input":"2022-09-19T06:21:10.08148Z","iopub.status.idle":"2022-09-19T06:21:10.090643Z","shell.execute_reply.started":"2022-09-19T06:21:10.081442Z","shell.execute_reply":"2022-09-19T06:21:10.089468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(2)","metadata":{"id":"AduqNAoXWsKm","outputId":"9261f0b0-e756-4c46-ae48-543cce7f2a25","execution":{"iopub.status.busy":"2022-09-19T06:21:11.408314Z","iopub.execute_input":"2022-09-19T06:21:11.409162Z","iopub.status.idle":"2022-09-19T06:21:11.423798Z","shell.execute_reply.started":"2022-09-19T06:21:11.409112Z","shell.execute_reply":"2022-09-19T06:21:11.4225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Story Generation and Visualization from Tweets**","metadata":{"id":"-KJ4-I_55Ctn"}},{"cell_type":"markdown","source":"* What are the most common words in the entire dataset?\n\n* What are the most common words in the dataset for negative and positive tweets, respectively?\n\n* How many hashtags are there in a tweet?\n\n* Which trends are associated with my dataset?\n\n* Which trends are associated with either of the sentiments? Are they compatible with the sentiments?","metadata":{"id":"dXD8GNBt5NTi"}},{"cell_type":"markdown","source":"A) **Understanding the common words used in the tweets: WordCloud**","metadata":{"id":"CdgCh-Qp5Zno"}},{"cell_type":"code","source":"df.head(2)","metadata":{"id":"9vUZvPVv5oiQ","outputId":"b96358b3-4e33-4365-b48e-e3a93f24f2aa","execution":{"iopub.status.busy":"2022-09-19T06:21:13.327516Z","iopub.execute_input":"2022-09-19T06:21:13.328255Z","iopub.status.idle":"2022-09-19T06:21:13.341513Z","shell.execute_reply.started":"2022-09-19T06:21:13.328214Z","shell.execute_reply":"2022-09-19T06:21:13.340332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create text from all tweets\nall_words = ' '.join([text for text in df['Tweet']])\n\nfrom wordcloud import WordCloud\nwordcloud = WordCloud(width=800, height=500, random_state=21, max_font_size=110).generate(all_words)\n\nplt.figure(figsize=(10, 7))\nplt.imshow(wordcloud, interpolation=\"bilinear\")\nplt.axis('off')\nplt.show()","metadata":{"id":"z1rnMsu_WUXx","outputId":"39b4030a-3844-4908-a76f-8ddef5e1a91e","execution":{"iopub.status.busy":"2022-09-19T06:21:13.895098Z","iopub.execute_input":"2022-09-19T06:21:13.895807Z","iopub.status.idle":"2022-09-19T06:21:20.129736Z","shell.execute_reply.started":"2022-09-19T06:21:13.895753Z","shell.execute_reply":"2022-09-19T06:21:20.128831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create text from just normal tweets\nnormal_words = ' '.join([text for text in df['Tweet'][df['Sentiment'] == 'Extremely Positive']])\n\nwordcloud = WordCloud(width=800, height=500, random_state=21, max_font_size=110).generate(normal_words)\nplt.figure(figsize=(10, 7))\nplt.imshow(wordcloud, interpolation=\"bilinear\")\nplt.axis('off')\nplt.show()","metadata":{"id":"wGLsco-KXnIJ","outputId":"9e7e982a-5115-4af0-ebd4-8995f227b684","execution":{"iopub.status.busy":"2022-09-19T06:21:20.132163Z","iopub.execute_input":"2022-09-19T06:21:20.13284Z","iopub.status.idle":"2022-09-19T06:21:22.184733Z","shell.execute_reply.started":"2022-09-19T06:21:20.132794Z","shell.execute_reply":"2022-09-19T06:21:22.183396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create text from just normal tweets\nnormal_words = ' '.join([text for text in df['Tweet'][df['Sentiment'] == 'Positive']])\n\nwordcloud = WordCloud(width=800, height=500, random_state=21, max_font_size=110).generate(normal_words)\nplt.figure(figsize=(10, 7))\nplt.imshow(wordcloud, interpolation=\"bilinear\")\nplt.axis('off')\nplt.show()","metadata":{"id":"SBJrqcRwWvS3","outputId":"499d1b86-cd34-48ae-8e7d-65d904abd1d0","execution":{"iopub.status.busy":"2022-09-19T06:21:22.186299Z","iopub.execute_input":"2022-09-19T06:21:22.18674Z","iopub.status.idle":"2022-09-19T06:21:24.77226Z","shell.execute_reply.started":"2022-09-19T06:21:22.186697Z","shell.execute_reply":"2022-09-19T06:21:24.771063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create text from just normal tweets\nnormal_words = ' '.join([text for text in df['Tweet'][df['Sentiment'] == 'Extremely Negative']])\n\nwordcloud = WordCloud(width=800, height=500, random_state=21, max_font_size=110).generate(normal_words)\nplt.figure(figsize=(10, 7))\nplt.imshow(wordcloud, interpolation=\"bilinear\")\nplt.axis('off')\nplt.show()","metadata":{"id":"rE0YoGBuW8Mg","outputId":"e700d61f-cb77-4083-84ba-1d66a8fc1d6e","execution":{"iopub.status.busy":"2022-09-19T06:21:24.775049Z","iopub.execute_input":"2022-09-19T06:21:24.776175Z","iopub.status.idle":"2022-09-19T06:21:26.607319Z","shell.execute_reply.started":"2022-09-19T06:21:24.776137Z","shell.execute_reply":"2022-09-19T06:21:26.606079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create text from just normal tweets\nnormal_words = ' '.join([text for text in df['Tweet'][df['Sentiment'] == 'Negative']])\n\nwordcloud = WordCloud(width=800, height=500, random_state=21, max_font_size=110).generate(normal_words)\nplt.figure(figsize=(10, 7))\nplt.imshow(wordcloud, interpolation=\"bilinear\")\nplt.axis('off')\nplt.show()","metadata":{"id":"VFvj7J38XW5C","outputId":"81540174-5c7c-4c75-e12e-f5738648439e","execution":{"iopub.status.busy":"2022-09-19T06:21:26.608629Z","iopub.execute_input":"2022-09-19T06:21:26.609078Z","iopub.status.idle":"2022-09-19T06:21:28.943467Z","shell.execute_reply.started":"2022-09-19T06:21:26.609032Z","shell.execute_reply":"2022-09-19T06:21:28.942335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create text from just normal tweets\nnormal_words = ' '.join([text for text in df['Tweet'][df['Sentiment'] == 'Neutral']])\n\nwordcloud = WordCloud(width=800, height=500, random_state=21, max_font_size=110).generate(normal_words)\nplt.figure(figsize=(10, 7))\nplt.imshow(wordcloud, interpolation=\"bilinear\")\nplt.axis('off')\nplt.show()","metadata":{"id":"qoOcpS6zaoNc","outputId":"ecaf5912-e375-44d1-9760-4f46fa52d71b","execution":{"iopub.status.busy":"2022-09-19T06:21:28.944848Z","iopub.execute_input":"2022-09-19T06:21:28.945201Z","iopub.status.idle":"2022-09-19T06:21:30.762264Z","shell.execute_reply.started":"2022-09-19T06:21:28.94517Z","shell.execute_reply":"2022-09-19T06:21:30.761013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**D) Understanding the impact of Hashtags on tweets sentiment**","metadata":{"id":"GCiDZQ4EX1P5"}},{"cell_type":"code","source":"# function to collect hashtags\ndef hashtag_extract(x):\n hashtags = []\n for i in x:\n ht = re.findall(r'#(\\w+)', i)\n hashtags.append(ht)\n return hashtags","metadata":{"id":"FAeBoSJmXfqj","execution":{"iopub.status.busy":"2022-09-19T06:21:30.764177Z","iopub.execute_input":"2022-09-19T06:21:30.764642Z","iopub.status.idle":"2022-09-19T06:21:30.770905Z","shell.execute_reply.started":"2022-09-19T06:21:30.764595Z","shell.execute_reply":"2022-09-19T06:21:30.77005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# extracting hashtags from non racist/sexist tweets\nHT_Extremely_Positive = hashtag_extract(df['OriginalTweet'][df['Sentiment'] == 'Extremely Positive'])\nHT_Positive = hashtag_extract(df['OriginalTweet'][df['Sentiment'] == 'Positive'])\nHT_Neutral = hashtag_extract(df['OriginalTweet'][df['Sentiment'] == 'Neutral'])\nHT_Negative = hashtag_extract(df['OriginalTweet'][df['Sentiment'] == 'Negative'])\nHT_Extremely_Negative = hashtag_extract(df['OriginalTweet'][df['Sentiment'] == 'Extremely Negative'])\n\n\n\n# unnesting list\nHT_Extremely_Positive = sum(HT_Extremely_Positive, [])\nHT_Positive = sum(HT_Positive, [])\nHT_Neutral = sum(HT_Neutral, [])\nHT_Negative = sum(HT_Negative,[])\nHT_Extremely_Negative = sum(HT_Extremely_Negative,[])","metadata":{"id":"9ANqduRcX-M_","execution":{"iopub.status.busy":"2022-09-19T06:21:30.772139Z","iopub.execute_input":"2022-09-19T06:21:30.772996Z","iopub.status.idle":"2022-09-19T06:21:33.304375Z","shell.execute_reply.started":"2022-09-19T06:21:30.772963Z","shell.execute_reply":"2022-09-19T06:21:33.303193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print(HT_Extremely_Positive)","metadata":{"id":"NHQLG1gIEkRi","outputId":"7be6bb18-5723-45e9-be78-57d594dbac28","execution":{"iopub.status.busy":"2022-09-19T06:21:33.305773Z","iopub.execute_input":"2022-09-19T06:21:33.306887Z","iopub.status.idle":"2022-09-19T06:21:33.317934Z","shell.execute_reply.started":"2022-09-19T06:21:33.306771Z","shell.execute_reply":"2022-09-19T06:21:33.316661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print(HT_Positive)","metadata":{"id":"1Vk3xneDG7sE","outputId":"bef6a51e-5530-4bd0-d332-14c0d04f76e8","execution":{"iopub.status.busy":"2022-09-19T06:21:33.321634Z","iopub.execute_input":"2022-09-19T06:21:33.322037Z","iopub.status.idle":"2022-09-19T06:21:33.335037Z","shell.execute_reply.started":"2022-09-19T06:21:33.321995Z","shell.execute_reply":"2022-09-19T06:21:33.333912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print(HT_Neutral)","metadata":{"id":"WhuPj1zAHCkd","outputId":"eeb4ca93-4118-49ac-8b0f-9b3d1473e065","execution":{"iopub.status.busy":"2022-09-19T06:21:33.336414Z","iopub.execute_input":"2022-09-19T06:21:33.336889Z","iopub.status.idle":"2022-09-19T06:21:33.352747Z","shell.execute_reply.started":"2022-09-19T06:21:33.336839Z","shell.execute_reply":"2022-09-19T06:21:33.351843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print(HT_Negative)","metadata":{"id":"EKKYTMtsHIfO","outputId":"4573992e-2a00-4a63-c021-aa8802b4ffaa","execution":{"iopub.status.busy":"2022-09-19T06:21:33.353644Z","iopub.execute_input":"2022-09-19T06:21:33.354197Z","iopub.status.idle":"2022-09-19T06:21:33.366414Z","shell.execute_reply.started":"2022-09-19T06:21:33.354152Z","shell.execute_reply":"2022-09-19T06:21:33.365285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print(HT_Extremely_Negative)","metadata":{"id":"f8y6Sys_HMA0","outputId":"64ad371f-f3f6-4de2-affa-45f506bbabd8","execution":{"iopub.status.busy":"2022-09-19T06:21:33.368079Z","iopub.execute_input":"2022-09-19T06:21:33.368503Z","iopub.status.idle":"2022-09-19T06:21:33.381012Z","shell.execute_reply.started":"2022-09-19T06:21:33.368462Z","shell.execute_reply":"2022-09-19T06:21:33.37984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# making frequency distribution top 10 Extremely Positive hashtags\na = nltk.FreqDist(HT_Extremely_Positive)\nd = pd.DataFrame({'Hashtag': list(a.keys()),\n 'Count' : list(a.values())})\n\nd = d.nlargest(columns = 'Count', n = 10)\n\nplt.figure(figsize = (16,5))\nax = sns.barplot(data =d, x = 'Hashtag', y = 'Count')\nplt.show()","metadata":{"id":"8cfDDpjSHPYC","outputId":"d326bd4f-b9d3-4ad8-cd43-a9a836347445","execution":{"iopub.status.busy":"2022-09-19T06:21:33.382205Z","iopub.execute_input":"2022-09-19T06:21:33.38291Z","iopub.status.idle":"2022-09-19T06:21:33.676334Z","shell.execute_reply.started":"2022-09-19T06:21:33.382872Z","shell.execute_reply":"2022-09-19T06:21:33.67533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# making frequency distribution top 10 Positive hashtags\na = nltk.FreqDist(HT_Positive)\nd = pd.DataFrame({'Hashtag': list(a.keys()),\n 'Count' : list(a.values())})\n\nd = d.nlargest(columns = 'Count', n = 10)\n\nplt.figure(figsize = (16,5))\nax = sns.barplot(data =d, x = 'Hashtag', y = 'Count')\nplt.show()","metadata":{"id":"83Li3KgIHlXv","outputId":"bedd41a7-eb50-44a8-ab53-95c0108a768e","execution":{"iopub.status.busy":"2022-09-19T06:21:33.677345Z","iopub.execute_input":"2022-09-19T06:21:33.67764Z","iopub.status.idle":"2022-09-19T06:21:33.97253Z","shell.execute_reply.started":"2022-09-19T06:21:33.677612Z","shell.execute_reply":"2022-09-19T06:21:33.971687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# making frequency distribution top 10 Positive hashtags\na = nltk.FreqDist(HT_Neutral)\nd = pd.DataFrame({'Hashtag': list(a.keys()),\n 'Count' : list(a.values())})\n\nd = d.nlargest(columns = 'Count', n = 10)\n\nplt.figure(figsize = (16,5))\nax = sns.barplot(data =d, x = 'Hashtag', y = 'Count')\nplt.show()","metadata":{"id":"41nxdDY-HoI-","outputId":"cd738a28-17f8-44df-a88e-4818552952fa","execution":{"iopub.status.busy":"2022-09-19T06:21:33.973747Z","iopub.execute_input":"2022-09-19T06:21:33.974264Z","iopub.status.idle":"2022-09-19T06:21:34.256521Z","shell.execute_reply.started":"2022-09-19T06:21:33.974232Z","shell.execute_reply":"2022-09-19T06:21:34.255178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# making frequency distribution top 10 Positive hashtags\na = nltk.FreqDist(HT_Negative)\nd = pd.DataFrame({'Hashtag': list(a.keys()),\n 'Count' : list(a.values())})\n\nd = d.nlargest(columns = 'Count', n = 10)\n\nplt.figure(figsize = (16,5))\nax = sns.barplot(data =d, x = 'Hashtag', y = 'Count')\nplt.show()","metadata":{"id":"rmEuERJ8H0t9","outputId":"57b53703-5269-4cce-9ae7-4ceeb7abc356","execution":{"iopub.status.busy":"2022-09-19T06:21:34.258197Z","iopub.execute_input":"2022-09-19T06:21:34.258928Z","iopub.status.idle":"2022-09-19T06:21:34.54377Z","shell.execute_reply.started":"2022-09-19T06:21:34.258845Z","shell.execute_reply":"2022-09-19T06:21:34.5426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{"id":"J4iUX7kLdYlx"}},{"cell_type":"code","source":"# making frequency distribution top 10 Extremely Negative hashtags\na = nltk.FreqDist(HT_Extremely_Negative)\nd = pd.DataFrame({'Hashtag': list(a.keys()),\n 'Count' : list(a.values())})\n\nd = d.nlargest(columns = 'Count', n = 10)\n\nplt.figure(figsize = (16,5))\nax = sns.barplot(data =d, x = 'Hashtag', y = 'Count')\nplt.show()","metadata":{"id":"Ym6fUvm2H5RI","outputId":"1ef791f3-ae73-4c8d-8654-fffb8a64d521","execution":{"iopub.status.busy":"2022-09-19T06:21:34.545297Z","iopub.execute_input":"2022-09-19T06:21:34.545637Z","iopub.status.idle":"2022-09-19T06:21:34.82723Z","shell.execute_reply.started":"2022-09-19T06:21:34.545605Z","shell.execute_reply":"2022-09-19T06:21:34.825949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#**4. Extracting Features from Cleaned Tweets**","metadata":{"id":"kOZWiNyDIPxc"}},{"cell_type":"code","source":"new_df = df[['Tweet','Sentiment']]","metadata":{"id":"fuhag8jiMKj2","execution":{"iopub.status.busy":"2022-09-19T06:21:36.145579Z","iopub.execute_input":"2022-09-19T06:21:36.145992Z","iopub.status.idle":"2022-09-19T06:21:36.158096Z","shell.execute_reply.started":"2022-09-19T06:21:36.145956Z","shell.execute_reply":"2022-09-19T06:21:36.157194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_df.head()","metadata":{"id":"-JAz-hazW-_7","outputId":"e3c3b9be-370e-4e06-95e0-78b8cb496144","execution":{"iopub.status.busy":"2022-09-19T06:21:36.922204Z","iopub.execute_input":"2022-09-19T06:21:36.923138Z","iopub.status.idle":"2022-09-19T06:21:36.935281Z","shell.execute_reply.started":"2022-09-19T06:21:36.923099Z","shell.execute_reply":"2022-09-19T06:21:36.933919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['Sentiment'].value_counts()","metadata":{"id":"wTrp1OB-dBwU","outputId":"34bc817b-4ef4-46f3-e1c4-422788bbe2b5","execution":{"iopub.status.busy":"2022-09-19T06:21:37.701321Z","iopub.execute_input":"2022-09-19T06:21:37.701738Z","iopub.status.idle":"2022-09-19T06:21:37.713667Z","shell.execute_reply.started":"2022-09-19T06:21:37.701703Z","shell.execute_reply":"2022-09-19T06:21:37.712226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Converting into lower case**","metadata":{"id":"ueDARf19YkyT"}},{"cell_type":"code","source":"#new_df[\"Tweet\"] = new_df[\"Tweet\"].str.lower()#.str.split()","metadata":{"id":"Lq-_RWaXXKfy","execution":{"iopub.status.busy":"2022-09-19T06:21:39.645351Z","iopub.execute_input":"2022-09-19T06:21:39.645773Z","iopub.status.idle":"2022-09-19T06:21:39.650652Z","shell.execute_reply.started":"2022-09-19T06:21:39.645727Z","shell.execute_reply":"2022-09-19T06:21:39.649272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#new_df.head(5)","metadata":{"id":"2lHEfK_mzyMV","execution":{"iopub.status.busy":"2022-09-19T06:21:40.282759Z","iopub.execute_input":"2022-09-19T06:21:40.283207Z","iopub.status.idle":"2022-09-19T06:21:40.288874Z","shell.execute_reply.started":"2022-09-19T06:21:40.283169Z","shell.execute_reply":"2022-09-19T06:21:40.287389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Removing Stopwords**","metadata":{"id":"_9BG3l6GYzJa"}},{"cell_type":"code","source":"nltk.download('stopwords')","metadata":{"id":"g-JX5_v30bGs","outputId":"efff600b-f366-4c20-806d-95a8f8f028e6","execution":{"iopub.status.busy":"2022-09-19T06:21:41.645633Z","iopub.execute_input":"2022-09-19T06:21:41.646375Z","iopub.status.idle":"2022-09-19T06:21:41.755217Z","shell.execute_reply.started":"2022-09-19T06:21:41.646334Z","shell.execute_reply":"2022-09-19T06:21:41.753938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from nltk.corpus import stopwords\nstop = stopwords.words('english')","metadata":{"id":"_TW2-bDxYjTM","execution":{"iopub.status.busy":"2022-09-19T06:21:44.188406Z","iopub.execute_input":"2022-09-19T06:21:44.189236Z","iopub.status.idle":"2022-09-19T06:21:44.197952Z","shell.execute_reply.started":"2022-09-19T06:21:44.189177Z","shell.execute_reply":"2022-09-19T06:21:44.196909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_df['Tweet'].apply(lambda x: [item for item in x if item not in stop])","metadata":{"id":"AKQ8AKs0Y3Ac","outputId":"0debeab1-6da7-4a07-cb6f-ca0ba8dd56ea","execution":{"iopub.status.busy":"2022-09-19T06:21:44.7868Z","iopub.execute_input":"2022-09-19T06:21:44.787234Z","iopub.status.idle":"2022-09-19T06:21:59.0646Z","shell.execute_reply.started":"2022-09-19T06:21:44.787198Z","shell.execute_reply":"2022-09-19T06:21:59.063369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_df.head(6)","metadata":{"id":"ErXF0fRgY993","outputId":"d1d2454a-1ac9-4309-dba7-788871645639","execution":{"iopub.status.busy":"2022-09-19T06:21:59.066913Z","iopub.execute_input":"2022-09-19T06:21:59.068373Z","iopub.status.idle":"2022-09-19T06:21:59.080256Z","shell.execute_reply.started":"2022-09-19T06:21:59.068318Z","shell.execute_reply":"2022-09-19T06:21:59.078939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_df.shape","metadata":{"id":"cwuGwh3y00go","outputId":"24dd1d82-9ef9-46e7-a343-fc9fa85bc8c6","execution":{"iopub.status.busy":"2022-09-19T06:21:59.081972Z","iopub.execute_input":"2022-09-19T06:21:59.082458Z","iopub.status.idle":"2022-09-19T06:21:59.093687Z","shell.execute_reply.started":"2022-09-19T06:21:59.082409Z","shell.execute_reply":"2022-09-19T06:21:59.092583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_df.isnull().sum()","metadata":{"id":"VRzxBp06ZE3H","outputId":"007b26ee-f093-4dd3-c006-2018773622d3","execution":{"iopub.status.busy":"2022-09-19T06:21:59.096515Z","iopub.execute_input":"2022-09-19T06:21:59.096919Z","iopub.status.idle":"2022-09-19T06:21:59.113494Z","shell.execute_reply.started":"2022-09-19T06:21:59.096869Z","shell.execute_reply":"2022-09-19T06:21:59.112229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_df.head(7)","metadata":{"id":"fOAeaMUs6LKx","outputId":"6628d6f6-e110-4446-884e-e3367f23ec09","execution":{"iopub.status.busy":"2022-09-19T06:21:59.115469Z","iopub.execute_input":"2022-09-19T06:21:59.115872Z","iopub.status.idle":"2022-09-19T06:21:59.130125Z","shell.execute_reply.started":"2022-09-19T06:21:59.115818Z","shell.execute_reply":"2022-09-19T06:21:59.129065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Spitting Our Dataset into Training And Testing Dataset ( For Multiclass Classification)","metadata":{"id":"xEGFCLQUCUhZ"}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ntrain,valid = train_test_split(new_df,test_size = 0.2,random_state=0,stratify = new_df.Sentiment.values) #stratification means that the train_test_split method returns training and test subsets that have the same proportions of class labels as the input dataset.\nprint(\"train shape : \", train.shape)\nprint(\"valid shape : \", valid.shape)","metadata":{"id":"94sKiacY6YhR","outputId":"c1ca3142-8497-43f6-deec-015298dceb2a","execution":{"iopub.status.busy":"2022-09-19T06:21:59.13164Z","iopub.execute_input":"2022-09-19T06:21:59.132309Z","iopub.status.idle":"2022-09-19T06:21:59.179825Z","shell.execute_reply.started":"2022-09-19T06:21:59.132273Z","shell.execute_reply":"2022-09-19T06:21:59.178591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Use Of Counter Vectorizer For Multi Class Classification","metadata":{"id":"rZIKvbDUCbIl"}},{"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer\nfrom nltk.corpus import stopwords\nstop = list(stopwords.words('english'))\nvectorizer = CountVectorizer(decode_error = 'replace',stop_words = stop)\n\nX_train = vectorizer.fit_transform(train.Tweet.values)\nX_valid = vectorizer.transform(valid.Tweet.values)\n\ny_train = train.Sentiment.values\ny_valid = valid.Sentiment.values\n\nprint(\"X_train.shape : \", X_train.shape)\nprint(\"X_train.shape : \", X_valid.shape)\nprint(\"y_train.shape : \", y_train.shape)\nprint(\"y_valid.shape : \", y_valid.shape)","metadata":{"id":"wQV3GlU36Ydz","outputId":"7a64c308-fbb0-4dab-8ed7-e2cd986c5d9a","execution":{"iopub.status.busy":"2022-09-19T06:21:59.181362Z","iopub.execute_input":"2022-09-19T06:21:59.182492Z","iopub.status.idle":"2022-09-19T06:22:00.178715Z","shell.execute_reply.started":"2022-09-19T06:21:59.18244Z","shell.execute_reply":"2022-09-19T06:22:00.177455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Naive Bayes Classifier for MULTICLASS Classification","metadata":{"id":"myV1IAbJsFRU"}},{"cell_type":"code","source":"from sklearn.naive_bayes import MultinomialNB\n\nnaiveByes_clf = MultinomialNB()\n\nnaiveByes_clf.fit(X_train,y_train)\n\nNB_prediction = naiveByes_clf.predict(X_valid)\nNB_accuracy = accuracy_score(y_valid,NB_prediction)\nprint(\"training accuracy Score : \",naiveByes_clf.score(X_train,y_train))\nprint(\"Validation accuracy Score : \",NB_accuracy )\nprint(classification_report(NB_prediction,y_valid))","metadata":{"id":"csVR9OLR7jpb","outputId":"449e2283-6243-4e67-d028-0a9b2a2eb65e","execution":{"iopub.status.busy":"2022-09-19T06:22:00.180245Z","iopub.execute_input":"2022-09-19T06:22:00.181013Z","iopub.status.idle":"2022-09-19T06:22:00.521262Z","shell.execute_reply.started":"2022-09-19T06:22:00.180965Z","shell.execute_reply":"2022-09-19T06:22:00.520303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Stochastic Gradient Descent-SGD Classifier( MULTICLASS CLASSIFICATION)","metadata":{"id":"js5YTX-8sf5s"}},{"cell_type":"code","source":"from sklearn.linear_model import SGDClassifier\n\nsgd_clf = SGDClassifier(loss = 'hinge', penalty = 'l2', random_state=0)\n\nsgd_clf.fit(X_train,y_train)\n\nsgd_prediction = sgd_clf.predict(X_valid)\nsgd_accuracy = accuracy_score(y_valid,sgd_prediction)\nprint(\"Training accuracy Score : \",sgd_clf.score(X_train,y_train))\nprint(\"Validation accuracy Score : \",sgd_accuracy )\nprint(classification_report(sgd_prediction,y_valid))","metadata":{"id":"yjnSTvWs7xkc","outputId":"24af7fc5-d2a5-4270-ba65-2369a8ad7442","execution":{"iopub.status.busy":"2022-09-19T06:22:00.522418Z","iopub.execute_input":"2022-09-19T06:22:00.523385Z","iopub.status.idle":"2022-09-19T06:22:01.934636Z","shell.execute_reply.started":"2022-09-19T06:22:00.523349Z","shell.execute_reply":"2022-09-19T06:22:01.933459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# RANDOM FOREST CLASSIFIER (For Multiclass Classification)","metadata":{"id":"c1ahAuzg-nER"}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\nrf_clf = RandomForestClassifier()\n\nrf_clf.fit(X_train,y_train)\n\nrf_prediction = rf_clf.predict(X_valid)\nrf_accuracy = accuracy_score(y_valid,rf_prediction)\nprint(\"Training accuracy Score : \",rf_clf.score(X_train,y_train))\nprint(\"Validation accuracy Score : \",rf_accuracy )\nprint(classification_report(rf_prediction,y_valid))","metadata":{"id":"_eZjPs1275_h","outputId":"765e39c9-9de6-4f3c-87a0-a34672946855","execution":{"iopub.status.busy":"2022-09-19T06:22:01.93858Z","iopub.execute_input":"2022-09-19T06:22:01.93908Z","iopub.status.idle":"2022-09-19T06:23:52.674935Z","shell.execute_reply.started":"2022-09-19T06:22:01.939044Z","shell.execute_reply":"2022-09-19T06:23:52.673833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Support vector machine( For Multiclass Classification)","metadata":{"id":"1FEbbbjq8_9V"}},{"cell_type":"code","source":"from sklearn.svm import SVC\n\nsvc = SVC()\n\nsvc.fit(X_train, y_train)\n\nsvc_prediction = svc.predict(X_valid)\nsvc_accuracy = accuracy_score(y_valid,svc_prediction)\nprint(\"Training accuracy Score : \",svc.score(X_train,y_train))\nprint(\"Validation accuracy Score : \",svc_accuracy )\nprint(classification_report(svc_prediction,y_valid))","metadata":{"id":"3aoQMTdu9DqW","outputId":"7ccc9132-cbe4-4dfb-8d63-8ba7eb30afa3","execution":{"iopub.status.busy":"2022-09-19T06:24:29.992459Z","iopub.execute_input":"2022-09-19T06:24:29.992842Z","iopub.status.idle":"2022-09-19T06:40:11.486489Z","shell.execute_reply.started":"2022-09-19T06:24:29.99281Z","shell.execute_reply":"2022-09-19T06:40:11.485257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Logistic Regression(For Multiclass Classification)","metadata":{"id":"jB-2Zpti_m0J"}},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nlogreg = LogisticRegression()\n\nlogreg.fit(X_train, y_train)\n\nlogreg_prediction = logreg.predict(X_valid)\nlogreg_accuracy = accuracy_score(y_valid,logreg_prediction)\nprint(\"Training accuracy Score : \",logreg.score(X_train,y_train))\nprint(\"Validation accuracy Score : \",logreg_accuracy )\nprint(classification_report(logreg_prediction,y_valid))","metadata":{"id":"y3LsCwfG_pym","outputId":"18eae539-305f-4676-da6b-c717efb20cce","execution":{"iopub.status.busy":"2022-09-19T07:01:47.619412Z","iopub.execute_input":"2022-09-19T07:01:47.61983Z","iopub.status.idle":"2022-09-19T07:01:56.864883Z","shell.execute_reply.started":"2022-09-19T07:01:47.619794Z","shell.execute_reply":"2022-09-19T07:01:56.863671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CATBOOST MODEL (For Multiclass Classification)","metadata":{"id":"4hQ6ya2Nf5L-"}},{"cell_type":"code","source":"!pip install catboost\n!pip install ipywidgets\n!jupyter nbextension enable --py widgetsnbextension","metadata":{"id":"VK4ls0BSk7fp","outputId":"03846b68-65fc-4240-bd70-b3b75cefd408","execution":{"iopub.status.busy":"2022-09-19T07:01:56.86682Z","iopub.execute_input":"2022-09-19T07:01:56.867182Z","iopub.status.idle":"2022-09-19T07:02:20.391571Z","shell.execute_reply.started":"2022-09-19T07:01:56.867152Z","shell.execute_reply":"2022-09-19T07:02:20.390104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from catboost import CatBoostClassifier, Pool, cv\nfrom sklearn.metrics import accuracy_score","metadata":{"id":"DHENvfS9lVv9","execution":{"iopub.status.busy":"2022-09-19T07:02:20.39342Z","iopub.execute_input":"2022-09-19T07:02:20.39382Z","iopub.status.idle":"2022-09-19T07:02:20.615702Z","shell.execute_reply.started":"2022-09-19T07:02:20.39378Z","shell.execute_reply":"2022-09-19T07:02:20.614786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clf2 = CatBoostClassifier()\n\n\nclf2.fit(X_train, y_train, \n eval_set=(X_valid, y_valid), \n verbose=False\n)\n\nprint('CatBoost model is fitted: ' + str(clf2.is_fitted()))\nprint('CatBoost model parameters:')\nprint(clf2.get_params())","metadata":{"id":"RX_eoPj5f34t","outputId":"8c8464bb-5892-42b0-f220-d1ff79dda5b2","execution":{"iopub.status.busy":"2022-09-19T07:02:20.618304Z","iopub.execute_input":"2022-09-19T07:02:20.618763Z","iopub.status.idle":"2022-09-19T07:06:18.697947Z","shell.execute_reply.started":"2022-09-19T07:02:20.618718Z","shell.execute_reply":"2022-09-19T07:06:18.696676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"catboost_prediction = clf2.predict(X_valid)\ncatboost_accuracy = accuracy_score(y_valid,catboost_prediction)\nprint(\"Training accuracy Score : \",clf2.score(X_train,y_train))\nprint(\"Validation accuracy Score : \",catboost_accuracy )\nprint(classification_report(catboost_prediction,y_valid))","metadata":{"id":"SG7YUXl1ppYx","outputId":"d0e149cc-05cb-4a92-c0de-f7248ab4f1be","execution":{"iopub.status.busy":"2022-09-19T07:06:18.699231Z","iopub.execute_input":"2022-09-19T07:06:18.699552Z","iopub.status.idle":"2022-09-19T07:06:19.605481Z","shell.execute_reply.started":"2022-09-19T07:06:18.699523Z","shell.execute_reply":"2022-09-19T07:06:19.603946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# All the multiclass models test accuracy in descending order","metadata":{"id":"D0iXzl6g_0Dr"}},{"cell_type":"code","source":"models = pd.DataFrame({\n 'Model': ['Support Vector Machines', 'Logistic Regression', \n 'Random Forest', 'Naive Bayes', \n 'Stochastic Gradient Decent','CatBoost'],\n 'Test accuracy': [svc_accuracy, logreg_accuracy, \n rf_accuracy, NB_accuracy, \n sgd_accuracy,catboost_accuracy]})\n\nmodels.sort_values(by='Test accuracy', ascending=False)","metadata":{"id":"dEpoZTfd_zLw","outputId":"3857259c-0561-4eac-91e9-11c22a341957","execution":{"iopub.status.busy":"2022-09-19T07:07:30.530525Z","iopub.execute_input":"2022-09-19T07:07:30.53099Z","iopub.status.idle":"2022-09-19T07:07:30.548093Z","shell.execute_reply.started":"2022-09-19T07:07:30.530951Z","shell.execute_reply":"2022-09-19T07:07:30.546821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **CONVRTING OUR MULTICLASS CLASSIFICATION INTO BINARY CLASSIFICATION**","metadata":{"id":"OcMRdgnwoKrU"}},{"cell_type":"code","source":"new_df.head()","metadata":{"id":"n2ibIt2N3U24","outputId":"6d33ce2f-f960-4a23-9d60-896acbc3139d","execution":{"iopub.status.busy":"2022-09-19T07:07:40.405703Z","iopub.execute_input":"2022-09-19T07:07:40.406123Z","iopub.status.idle":"2022-09-19T07:07:40.417683Z","shell.execute_reply.started":"2022-09-19T07:07:40.406088Z","shell.execute_reply":"2022-09-19T07:07:40.416297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CATBOOST = new_df[['Tweet','Sentiment']]","metadata":{"id":"LeidxB9myDd4","execution":{"iopub.status.busy":"2022-09-19T07:07:42.04521Z","iopub.execute_input":"2022-09-19T07:07:42.046345Z","iopub.status.idle":"2022-09-19T07:07:42.06195Z","shell.execute_reply.started":"2022-09-19T07:07:42.046298Z","shell.execute_reply":"2022-09-19T07:07:42.060358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CATBOOST.head(2)","metadata":{"id":"kx9ouKbm1Hwp","outputId":"0f15a07d-c6e2-410f-bc35-c56e22c19792","execution":{"iopub.status.busy":"2022-09-19T07:07:42.525479Z","iopub.execute_input":"2022-09-19T07:07:42.526145Z","iopub.status.idle":"2022-09-19T07:07:42.537599Z","shell.execute_reply.started":"2022-09-19T07:07:42.526109Z","shell.execute_reply":"2022-09-19T07:07:42.535956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CATBOOST[\"Sentiment\"]= CATBOOST[\"Sentiment\"].replace('Positive',1) \nCATBOOST[\"Sentiment\"]= CATBOOST[\"Sentiment\"].replace('Extremely Positive',1) \nCATBOOST[\"Sentiment\"]= CATBOOST[\"Sentiment\"].replace('Neutral',1) \nCATBOOST[\"Sentiment\"]= CATBOOST[\"Sentiment\"].replace('Negative',0) \nCATBOOST[\"Sentiment\"]= CATBOOST[\"Sentiment\"].replace('Extremely Negative',0) \n","metadata":{"id":"kAQHjxkI1Ot3","execution":{"iopub.status.busy":"2022-09-19T07:07:42.995375Z","iopub.execute_input":"2022-09-19T07:07:42.996626Z","iopub.status.idle":"2022-09-19T07:07:43.027601Z","shell.execute_reply.started":"2022-09-19T07:07:42.996575Z","shell.execute_reply":"2022-09-19T07:07:43.026676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CATBOOST.head(2)","metadata":{"id":"naV50ws_4XTG","outputId":"5f960c5c-fde2-49fa-989f-a9e4bbf82bd9","execution":{"iopub.status.busy":"2022-09-19T07:07:43.640912Z","iopub.execute_input":"2022-09-19T07:07:43.642235Z","iopub.status.idle":"2022-09-19T07:07:43.652421Z","shell.execute_reply.started":"2022-09-19T07:07:43.642183Z","shell.execute_reply":"2022-09-19T07:07:43.651433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CATBOOST['Sentiment'].value_counts()","metadata":{"id":"mJQO-DuL4oWb","outputId":"da0972f6-8151-42ac-bd0c-296a644b1c5b","execution":{"iopub.status.busy":"2022-09-19T07:07:44.172977Z","iopub.execute_input":"2022-09-19T07:07:44.173412Z","iopub.status.idle":"2022-09-19T07:07:44.184331Z","shell.execute_reply.started":"2022-09-19T07:07:44.173376Z","shell.execute_reply":"2022-09-19T07:07:44.183082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = CATBOOST.drop('Sentiment', axis=1)\ny = CATBOOST.Sentiment","metadata":{"id":"ZOECBBDR1_yL","execution":{"iopub.status.busy":"2022-09-19T07:07:44.720287Z","iopub.execute_input":"2022-09-19T07:07:44.721094Z","iopub.status.idle":"2022-09-19T07:07:44.7294Z","shell.execute_reply.started":"2022-09-19T07:07:44.72105Z","shell.execute_reply":"2022-09-19T07:07:44.728116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"_wRBE9Sa7g6q"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nltk.download('stopwords')","metadata":{"id":"aU2Qvn027hNu","outputId":"6c1c9649-6413-4c48-aede-d23744d6c02d","execution":{"iopub.status.busy":"2022-09-19T07:07:45.830564Z","iopub.execute_input":"2022-09-19T07:07:45.831093Z","iopub.status.idle":"2022-09-19T07:07:45.84158Z","shell.execute_reply.started":"2022-09-19T07:07:45.83105Z","shell.execute_reply":"2022-09-19T07:07:45.840347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from nltk.corpus import stopwords\nstop = stopwords.words('english')","metadata":{"id":"Wm6Kr9Ia7hNw","execution":{"iopub.status.busy":"2022-09-19T07:07:46.341695Z","iopub.execute_input":"2022-09-19T07:07:46.342127Z","iopub.status.idle":"2022-09-19T07:07:46.347663Z","shell.execute_reply.started":"2022-09-19T07:07:46.342088Z","shell.execute_reply":"2022-09-19T07:07:46.346453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CATBOOST['Tweet'].apply(lambda x: [item for item in x if item not in stop])","metadata":{"id":"cDiOWxNk7hNx","outputId":"03d47964-9dbd-4db3-cc73-a54ee95e1912","execution":{"iopub.status.busy":"2022-09-19T07:07:46.755847Z","iopub.execute_input":"2022-09-19T07:07:46.756291Z","iopub.status.idle":"2022-09-19T07:08:01.224647Z","shell.execute_reply.started":"2022-09-19T07:07:46.756253Z","shell.execute_reply":"2022-09-19T07:08:01.223293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CATBOOST.head(2)","metadata":{"id":"3PmzyGYH7pKj","outputId":"6ea56b9f-6de1-4711-c695-e549c5c6b9bb","execution":{"iopub.status.busy":"2022-09-19T07:08:01.22725Z","iopub.execute_input":"2022-09-19T07:08:01.228071Z","iopub.status.idle":"2022-09-19T07:08:01.238613Z","shell.execute_reply.started":"2022-09-19T07:08:01.228013Z","shell.execute_reply":"2022-09-19T07:08:01.23767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# DIVIDING OUR DATASET INTO TRAINING AND TESTING","metadata":{"id":"wL1GXFZJAWLK"}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ntrain,valid = train_test_split(CATBOOST,test_size = 0.2,random_state=0,stratify = CATBOOST.Sentiment.values) #stratification means that the train_test_split method returns training and test subsets that have the same proportions of class labels as the input dataset.\nprint(\"train shape : \", train.shape)\nprint(\"valid shape : \", valid.shape)","metadata":{"id":"SYoajP7J7hNx","outputId":"7e66ba5b-e258-42cc-a613-f68eac91a9b5","execution":{"iopub.status.busy":"2022-09-19T07:08:01.239745Z","iopub.execute_input":"2022-09-19T07:08:01.240138Z","iopub.status.idle":"2022-09-19T07:08:01.269697Z","shell.execute_reply.started":"2022-09-19T07:08:01.240107Z","shell.execute_reply":"2022-09-19T07:08:01.268416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# USING COUNT VECTORIZER","metadata":{"id":"aJmIJhUeAeKx"}},{"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer\nfrom nltk.corpus import stopwords\nstop = list(stopwords.words('english'))\nvectorizer = CountVectorizer(decode_error = 'replace',stop_words = stop)\n\nX_train = vectorizer.fit_transform(train.Tweet.values)\nX_valid = vectorizer.transform(valid.Tweet.values)\n\ny_train = train.Sentiment.values\ny_valid = valid.Sentiment.values\n\nprint(\"X_train.shape : \", X_train.shape)\nprint(\"X_train.shape : \", X_valid.shape)\nprint(\"y_train.shape : \", y_train.shape)\nprint(\"y_valid.shape : \", y_valid.shape)","metadata":{"id":"70R_BSR57hNy","outputId":"96037182-d05e-4302-ca00-7842bcb5a0fb","execution":{"iopub.status.busy":"2022-09-19T07:08:01.272271Z","iopub.execute_input":"2022-09-19T07:08:01.272622Z","iopub.status.idle":"2022-09-19T07:08:02.309306Z","shell.execute_reply.started":"2022-09-19T07:08:01.27259Z","shell.execute_reply":"2022-09-19T07:08:02.308123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# NAIVE BAYES CLASSIFIER FOR BINARY CLASSIFICATION.","metadata":{"id":"9BT6gcJvAIAK"}},{"cell_type":"code","source":"from sklearn.naive_bayes import MultinomialNB\n\nnaiveByes_clf = MultinomialNB()\n\nnaiveByes_clf.fit(X_train,y_train)\n\nNB_prediction = naiveByes_clf.predict(X_valid)\nNB_accuracy = accuracy_score(y_valid,NB_prediction)\nprint(\"training accuracy Score : \",naiveByes_clf.score(X_train,y_train))\nprint(\"Validation accuracy Score : \",NB_accuracy )\nprint(classification_report(NB_prediction,y_valid))","metadata":{"id":"hGzzqlFp7hNz","outputId":"8cd42565-2af2-43eb-881a-584413ca687c","execution":{"iopub.status.busy":"2022-09-19T07:08:02.310532Z","iopub.execute_input":"2022-09-19T07:08:02.310893Z","iopub.status.idle":"2022-09-19T07:08:02.361978Z","shell.execute_reply.started":"2022-09-19T07:08:02.310847Z","shell.execute_reply":"2022-09-19T07:08:02.360653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# RANDOM FOREST CLASSIFIER FOR BINARY CLASSIFICATION","metadata":{"id":"aUfi0FdR__h5"}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\nrf_clf = RandomForestClassifier()\n\nrf_clf.fit(X_train,y_train)\n\nrf_prediction = rf_clf.predict(X_valid)\nrf_accuracy = accuracy_score(y_valid,rf_prediction)\nprint(\"Training accuracy Score : \",rf_clf.score(X_train,y_train))\nprint(\"Validation accuracy Score : \",rf_accuracy )\nprint(classification_report(rf_prediction,y_valid))","metadata":{"id":"V3L3LaHx8YxG","outputId":"6b427612-aeaa-4419-a9c9-00a9974f5ca8","execution":{"iopub.status.busy":"2022-09-19T07:08:02.364687Z","iopub.execute_input":"2022-09-19T07:08:02.365574Z","iopub.status.idle":"2022-09-19T07:09:17.395606Z","shell.execute_reply.started":"2022-09-19T07:08:02.365535Z","shell.execute_reply":"2022-09-19T07:09:17.394247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# LOGISTIC REGRESSION(BINARY CLASSIFICATION)","metadata":{"id":"to6Yup8Y_3Ng"}},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nlogreg = LogisticRegression()\n\nlogreg.fit(X_train, y_train)\n\nlogreg_prediction = logreg.predict(X_valid)\nlogreg_accuracy = accuracy_score(y_valid,logreg_prediction)\nprint(\"Training accuracy Score : \",logreg.score(X_train,y_train))\nprint(\"Validation accuracy Score : \",logreg_accuracy )\nprint(classification_report(logreg_prediction,y_valid))","metadata":{"id":"S5VrQ1Wo8Zgx","outputId":"b284b0d5-18f3-4167-951b-2b442081a43a","execution":{"iopub.status.busy":"2022-09-19T07:09:17.397145Z","iopub.execute_input":"2022-09-19T07:09:17.39833Z","iopub.status.idle":"2022-09-19T07:09:19.812881Z","shell.execute_reply.started":"2022-09-19T07:09:17.398286Z","shell.execute_reply":"2022-09-19T07:09:19.811293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CATBOOST ALGORITHM FOR BINARY CLASSIFICATION","metadata":{"id":"sCRvQXQY_rQ4"}},{"cell_type":"code","source":"clf2 = CatBoostClassifier()\n\n\nclf2.fit(X_train, y_train, \n eval_set=(X_valid, y_valid), \n verbose=False\n)\n\nprint('CatBoost model is fitted: ' + str(clf2.is_fitted()))\nprint('CatBoost model parameters:')\nprint(clf2.get_params())","metadata":{"id":"s9E5PVRJ9B-h","outputId":"1fc55d7a-0d10-41f0-d613-88d2c0015c00","execution":{"iopub.status.busy":"2022-09-19T07:09:19.814905Z","iopub.execute_input":"2022-09-19T07:09:19.817018Z","iopub.status.idle":"2022-09-19T07:10:41.011416Z","shell.execute_reply.started":"2022-09-19T07:09:19.816954Z","shell.execute_reply":"2022-09-19T07:10:41.010167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"catboost_prediction = clf2.predict(X_valid)\ncatboost_accuracy = accuracy_score(y_valid,catboost_prediction)\nprint(\"Training accuracy Score : \",clf2.score(X_train,y_train))\nprint(\"Validation accuracy Score : \",catboost_accuracy )\nprint(classification_report(catboost_prediction,y_valid))","metadata":{"id":"lbCR3AfF9Vy4","outputId":"8caaf881-aaa3-4eea-baa1-c18f53226609","execution":{"iopub.status.busy":"2022-09-19T07:10:41.012998Z","iopub.execute_input":"2022-09-19T07:10:41.014115Z","iopub.status.idle":"2022-09-19T07:10:41.343183Z","shell.execute_reply.started":"2022-09-19T07:10:41.014078Z","shell.execute_reply":"2022-09-19T07:10:41.342076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# XG BOOST( BINARY CLASSIFICATION)","metadata":{"id":"pZjoXIL8-vBb"}},{"cell_type":"code","source":"#takes huge amount of time to execute\nimport xgboost as xgb\n\nxgboost_clf = xgb.XGBClassifier()\n\nxgboost_clf.fit(X_train, y_train)\n\nxgb_prediction = xgboost_clf.predict(X_valid)\nxgb_accuracy = accuracy_score(y_valid,xgb_prediction)\nprint(\"Training accuracy Score : \",xgboost_clf.score(X_train,y_train))\nprint(\"Validation accuracy Score : \",xgb_accuracy )\nprint(classification_report(xgb_prediction,y_valid))","metadata":{"id":"Fe-OiRgI-U91","outputId":"2cdea2bc-c7a1-4fea-8c65-90940b2a0d3c","execution":{"iopub.status.busy":"2022-09-19T07:10:41.347027Z","iopub.execute_input":"2022-09-19T07:10:41.347397Z","iopub.status.idle":"2022-09-19T07:10:45.136069Z","shell.execute_reply.started":"2022-09-19T07:10:41.347356Z","shell.execute_reply":"2022-09-19T07:10:45.134654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# SUPPORT VECTOR MACHINE(BINARY CLASSIFICATION)","metadata":{"id":"xd97J4mU_C0u"}},{"cell_type":"code","source":"from sklearn.svm import SVC\n\nsvc = SVC()\n\nsvc.fit(X_train, y_train)\n\nsvc_prediction = svc.predict(X_valid)\nsvc_accuracy = accuracy_score(y_valid,svc_prediction)\nprint(\"Training accuracy Score : \",svc.score(X_train,y_train))\nprint(\"Validation accuracy Score : \",svc_accuracy )\nprint(classification_report(svc_prediction,y_valid))","metadata":{"id":"Yij2VGJD-00y","outputId":"feff1030-0952-4267-d1fb-7f4b01de6506","execution":{"iopub.status.busy":"2022-09-19T07:10:45.137545Z","iopub.execute_input":"2022-09-19T07:10:45.138048Z","iopub.status.idle":"2022-09-19T07:25:11.102532Z","shell.execute_reply.started":"2022-09-19T07:10:45.137999Z","shell.execute_reply":"2022-09-19T07:25:11.101175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Winner Model:**","metadata":{"id":"o4762Q92RHBM"}},{"cell_type":"markdown","source":"# Stochastic Gradient Descent-SGD Classifier( BINARY CLASSIFICATION)","metadata":{"id":"qjmGmxMKBq8T"}},{"cell_type":"code","source":"from sklearn.linear_model import SGDClassifier\n\nsgd_clf = SGDClassifier(loss = 'hinge', penalty = 'l2', random_state=0)\n\nsgd_clf.fit(X_train,y_train)\n\nsgd_prediction = sgd_clf.predict(X_valid)\nsgd_accuracy = accuracy_score(y_valid,sgd_prediction)\nprint(\"Training accuracy Score : \",sgd_clf.score(X_train,y_train))\nprint(\"Validation accuracy Score : \",sgd_accuracy )\nprint(classification_report(sgd_prediction,y_valid))","metadata":{"id":"xbGcrWUOB22L","outputId":"673afdf3-252e-4566-fa87-fd20eecff1a0","execution":{"iopub.status.busy":"2022-09-19T07:25:11.104114Z","iopub.execute_input":"2022-09-19T07:25:11.104435Z","iopub.status.idle":"2022-09-19T07:25:11.381996Z","shell.execute_reply.started":"2022-09-19T07:25:11.104407Z","shell.execute_reply":"2022-09-19T07:25:11.380621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the predicted classes\ntrain_class_preds = sgd_clf.predict(X_train)\ntest_class_preds = sgd_clf.predict(X_valid)","metadata":{"id":"h1s4fck2HVKI","execution":{"iopub.status.busy":"2022-09-19T07:25:11.384763Z","iopub.execute_input":"2022-09-19T07:25:11.385128Z","iopub.status.idle":"2022-09-19T07:25:11.393455Z","shell.execute_reply.started":"2022-09-19T07:25:11.385097Z","shell.execute_reply":"2022-09-19T07:25:11.392149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the confusion matrix for both train and test. We are getting very low type and type 2 errors.\n\nlabels = ['Negative', 'Positive']\ncm = confusion_matrix(y_train, train_class_preds)\nprint(cm)\n\nax= plt.subplot()\nsns.heatmap(cm, annot=True, ax = ax) #annot=True to annotate cells\n\n# labels, title and ticks\nax.set_xlabel('Predicted labels')\nax.set_ylabel('True labels')\nax.set_title('Confusion Matrix')\nax.xaxis.set_ticklabels(labels)\nax.yaxis.set_ticklabels(labels)","metadata":{"id":"j3P13pc7HLXM","outputId":"2ae327b5-7e94-4589-ff89-b6a2e444bb41","execution":{"iopub.status.busy":"2022-09-19T07:25:11.394816Z","iopub.execute_input":"2022-09-19T07:25:11.395268Z","iopub.status.idle":"2022-09-19T07:25:11.702203Z","shell.execute_reply.started":"2022-09-19T07:25:11.395236Z","shell.execute_reply":"2022-09-19T07:25:11.700869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's check the overall accuracy. Overall accuracy is very good.\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import log_loss\n\ny_pred = sgd_clf.predict(X_valid)\n\nscore =accuracy_score(y_valid,y_pred)\nprint('accuracy is', score)","metadata":{"id":"NiehoHixI38s","outputId":"60ec5d60-d28b-41bd-ddaf-94e87ce76798","execution":{"iopub.status.busy":"2022-09-19T07:25:11.703509Z","iopub.execute_input":"2022-09-19T07:25:11.704743Z","iopub.status.idle":"2022-09-19T07:25:11.712772Z","shell.execute_reply.started":"2022-09-19T07:25:11.704708Z","shell.execute_reply":"2022-09-19T07:25:11.711654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# F1 score for our classifier\nfrom sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score, classification_report, confusion_matrix\n\n\ny_pred = sgd_clf.predict(X_valid)\nprint(f1_score(y_valid,y_pred, average=\"macro\"))","metadata":{"id":"kPiPlY8oJHrS","outputId":"417fde11-9523-4672-ab90-294628c342f7","execution":{"iopub.status.busy":"2022-09-19T07:25:11.714103Z","iopub.execute_input":"2022-09-19T07:25:11.714881Z","iopub.status.idle":"2022-09-19T07:25:11.727181Z","shell.execute_reply.started":"2022-09-19T07:25:11.714825Z","shell.execute_reply":"2022-09-19T07:25:11.72593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#score is mean accuracy\nscikit_score = sgd_clf.score(X_valid,y_valid)\nprint('scikit score:', scikit_score)","metadata":{"id":"8aWTQfKmJahz","outputId":"6937cfb2-6575-4d06-dfdb-4b65ef5cf8ab","execution":{"iopub.status.busy":"2022-09-19T07:25:11.728427Z","iopub.execute_input":"2022-09-19T07:25:11.730011Z","iopub.status.idle":"2022-09-19T07:25:11.739548Z","shell.execute_reply.started":"2022-09-19T07:25:11.729969Z","shell.execute_reply":"2022-09-19T07:25:11.738391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Recall score for our winner model\nrecall_score(y_valid, y_pred, average='macro')","metadata":{"id":"02PTTDJfJicM","outputId":"e33e64c3-f0fc-4a4a-ad16-9f91ad667dc6","execution":{"iopub.status.busy":"2022-09-19T07:25:11.740851Z","iopub.execute_input":"2022-09-19T07:25:11.741571Z","iopub.status.idle":"2022-09-19T07:25:11.755017Z","shell.execute_reply.started":"2022-09-19T07:25:11.741538Z","shell.execute_reply":"2022-09-19T07:25:11.753814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Classification Report for our stochastic gradient descent algorithm\nclassification_report(y_valid,y_pred)","metadata":{"id":"W9btosFcKIaC","outputId":"1b7153c4-7292-476b-e742-558443cb83c8","execution":{"iopub.status.busy":"2022-09-19T07:25:11.756898Z","iopub.execute_input":"2022-09-19T07:25:11.757473Z","iopub.status.idle":"2022-09-19T07:25:11.783112Z","shell.execute_reply.started":"2022-09-19T07:25:11.757438Z","shell.execute_reply":"2022-09-19T07:25:11.781902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Very low type 1 and type 2 error\nconfusion_matrix(y_valid,y_pred)","metadata":{"id":"RWzN3zFhKbXg","outputId":"6f520fc0-20e6-4f5e-c7a0-6213af5f26d3","execution":{"iopub.status.busy":"2022-09-19T07:25:11.784541Z","iopub.execute_input":"2022-09-19T07:25:11.784976Z","iopub.status.idle":"2022-09-19T07:25:11.795174Z","shell.execute_reply.started":"2022-09-19T07:25:11.784933Z","shell.execute_reply":"2022-09-19T07:25:11.794033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EVALUATION OF ALL BINARY CLASSIFICATION MODELS\n\n**All the model test accuracy by descending order**","metadata":{"id":"008mcsIB_eJl"}},{"cell_type":"code","source":"models = pd.DataFrame({\n 'Model': ['Support Vector Machines', 'Logistic Regression', \n 'Random Forest', 'Naive Bayes', \n 'Stochastic Gradient Decent', 'XGBoost','CatBoost'],\n 'Test accuracy': [svc_accuracy, logreg_accuracy, \n rf_accuracy, NB_accuracy, \n sgd_accuracy, xgb_accuracy,catboost_accuracy]})\n\nmodels.sort_values(by='Test accuracy', ascending=False)","metadata":{"id":"OycbTt1K_KZC","outputId":"2339f165-77c0-4fcc-b18a-48f906dfbb6d","execution":{"iopub.status.busy":"2022-09-19T07:25:11.796362Z","iopub.execute_input":"2022-09-19T07:25:11.796698Z","iopub.status.idle":"2022-09-19T07:25:11.811709Z","shell.execute_reply.started":"2022-09-19T07:25:11.796668Z","shell.execute_reply":"2022-09-19T07:25:11.810753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **THANKS YOU. WHAT'S YOUR SENTIMENT FOR THIS PROJECT? FEELING HAPPY OR NOT??**","metadata":{"id":"2RAqOQdlFQyf"}},{"cell_type":"markdown","source":"","metadata":{"id":"-5PA64nlFLUc"}}]}