{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Text Classification with Bag of Words - Natural Language Processing\n\n![](https://i.imgur.com/hlEQ5X8.png)\n\n> _\"Natural language processing (NLP) is a subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data.\"_ - Wikipedia\n\n> _**Bag of Words**: The bag-of-words (BOW) model is a representation that turns arbitrary text into fixed-length vectors by counting how many times each word appears._\n\nOutline:\n\n1. Download and explore a real-world dataset\n2. Apply text preprocessing techniques\n3. Implement the bag of words model\n4. Train ML models for text classification\n5. Make predictions and submit to Kaggle\n\n\nDataset: https://www.kaggle.com/c/quora-insincere-questions-classification\n\n","metadata":{"id":"tBxC0flTQDd3"}},{"cell_type":"markdown","source":"## Download and Explore the Data\n\nOutline:\n\n1. Download the dataset from Kaggle to Colab\n2. Explore the data using Pandas\n3. Create a small working sample","metadata":{"id":"ZOcWwlvxQoSJ"}},{"cell_type":"markdown","source":"### Download the Data to Colab\n\nUpload your `kaggle.json` to Colab. Get it here: https://www.kaggle.com/docs/api#authentication\n\n","metadata":{"id":"xCBkYxdzRPPq"}},{"cell_type":"code","source":"# !kaggle competitions download -c quora-insincere-questions-classification -f train.csv -p data","metadata":{"executionInfo":{"elapsed":5381,"status":"ok","timestamp":1692965483285,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"Y-gnbjWgRg13","outputId":"552e83b0-f7ea-43d7-add6-356dce722a83","execution":{"iopub.status.busy":"2023-08-25T15:55:30.280683Z","iopub.execute_input":"2023-08-25T15:55:30.281058Z","iopub.status.idle":"2023-08-25T15:55:30.305656Z","shell.execute_reply.started":"2023-08-25T15:55:30.281028Z","shell.execute_reply":"2023-08-25T15:55:30.304861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import os\n# os.environ['KAGGLE_CONFIG_DIR'] = '.'","metadata":{"executionInfo":{"elapsed":9,"status":"ok","timestamp":1692965483285,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"ADfR5g8MSZ0_","execution":{"iopub.status.busy":"2023-08-25T15:55:30.307532Z","iopub.execute_input":"2023-08-25T15:55:30.307887Z","iopub.status.idle":"2023-08-25T15:55:30.311938Z","shell.execute_reply.started":"2023-08-25T15:55:30.307855Z","shell.execute_reply":"2023-08-25T15:55:30.311090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls","metadata":{"executionInfo":{"elapsed":10,"status":"ok","timestamp":1692965483286,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"ixLiVhlOq7pr","outputId":"210cb6bf-68b3-4e2e-db02-44d4337eb5b6","execution":{"iopub.status.busy":"2023-08-25T15:55:30.312960Z","iopub.execute_input":"2023-08-25T15:55:30.314115Z","iopub.status.idle":"2023-08-25T15:55:31.332683Z","shell.execute_reply.started":"2023-08-25T15:55:30.314081Z","shell.execute_reply":"2023-08-25T15:55:31.331021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !kaggle competitions download -c quora-insincere-questions-classification -f test.csv -p data\n# !kaggle competitions download -c quora-insincere-questions-classification -f sample_submission.csv -p data","metadata":{"executionInfo":{"elapsed":5965,"status":"ok","timestamp":1692965489245,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"Wizvk8maq546","outputId":"3f823ee3-6dd1-4bd0-9a1c-7ae90fa837f7","execution":{"iopub.status.busy":"2023-08-25T15:55:31.334566Z","iopub.execute_input":"2023-08-25T15:55:31.335106Z","iopub.status.idle":"2023-08-25T15:55:31.340551Z","shell.execute_reply.started":"2023-08-25T15:55:31.335066Z","shell.execute_reply":"2023-08-25T15:55:31.339333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Explore the Data using Pandas","metadata":{"id":"Ql6Gv4shTSBJ"}},{"cell_type":"code","source":"data_dir = '../input/quora-insincere-questions-classification'\ntrain_fname = data_dir + '/train.csv'\ntest_fname = data_dir +'/test.csv'\nsample_fname = data_dir +'/sample_submission.csv'","metadata":{"execution":{"iopub.status.busy":"2023-08-25T15:55:31.343151Z","iopub.execute_input":"2023-08-25T15:55:31.344625Z","iopub.status.idle":"2023-08-25T15:55:31.354077Z","shell.execute_reply.started":"2023-08-25T15:55:31.344585Z","shell.execute_reply":"2023-08-25T15:55:31.352734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_fname = 'data/train.csv.zip'\n# test_fname = 'data/test.csv.zip'\n# sample_fname = 'data/sample_submission.csv.zip'","metadata":{"executionInfo":{"elapsed":6,"status":"ok","timestamp":1692965489246,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"pf65NRzdQ-hU","execution":{"iopub.status.busy":"2023-08-25T15:55:31.355446Z","iopub.execute_input":"2023-08-25T15:55:31.355835Z","iopub.status.idle":"2023-08-25T15:55:31.368459Z","shell.execute_reply.started":"2023-08-25T15:55:31.355803Z","shell.execute_reply":"2023-08-25T15:55:31.367246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nraw_df = pd.read_csv(train_fname)","metadata":{"executionInfo":{"elapsed":3934,"status":"ok","timestamp":1692965493175,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"H1FjN5gDTfUY","execution":{"iopub.status.busy":"2023-08-25T15:55:31.370255Z","iopub.execute_input":"2023-08-25T15:55:31.370623Z","iopub.status.idle":"2023-08-25T15:55:36.847393Z","shell.execute_reply.started":"2023-08-25T15:55:31.370592Z","shell.execute_reply":"2023-08-25T15:55:36.846155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"raw_df","metadata":{"executionInfo":{"elapsed":46,"status":"ok","timestamp":1692965493176,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"CHMgt_5iTfRt","outputId":"8f465eb7-4832-4f22-f682-b714bd1ce7ac","execution":{"iopub.status.busy":"2023-08-25T15:55:36.851564Z","iopub.execute_input":"2023-08-25T15:55:36.851955Z","iopub.status.idle":"2023-08-25T15:55:36.880325Z","shell.execute_reply.started":"2023-08-25T15:55:36.851926Z","shell.execute_reply":"2023-08-25T15:55:36.879191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sincere_df = raw_df[raw_df.target == 0]\nsincere_df.question_text.values[:5]","metadata":{"executionInfo":{"elapsed":45,"status":"ok","timestamp":1692965493177,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"wV2xGZ3STfPx","outputId":"a9d8ead9-fc67-4eb7-aee9-e0e6edb2e5af","execution":{"iopub.status.busy":"2023-08-25T15:55:36.881556Z","iopub.execute_input":"2023-08-25T15:55:36.881913Z","iopub.status.idle":"2023-08-25T15:55:36.984207Z","shell.execute_reply.started":"2023-08-25T15:55:36.881883Z","shell.execute_reply":"2023-08-25T15:55:36.983129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"insincere_df = raw_df[raw_df.target == 1]\ninsincere_df.question_text.values[:5]","metadata":{"executionInfo":{"elapsed":36,"status":"ok","timestamp":1692965493178,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"me4srOFdUMCQ","outputId":"aa337f2b-7c3e-46c2-bf90-6e501ba5b076","execution":{"iopub.status.busy":"2023-08-25T15:55:36.985994Z","iopub.execute_input":"2023-08-25T15:55:36.986555Z","iopub.status.idle":"2023-08-25T15:55:37.013048Z","shell.execute_reply.started":"2023-08-25T15:55:36.986506Z","shell.execute_reply":"2023-08-25T15:55:37.012055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"raw_df.target.value_counts(normalize=True)","metadata":{"executionInfo":{"elapsed":31,"status":"ok","timestamp":1692965493178,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"cTApBtxIUMAJ","outputId":"92ca391a-f6bd-4132-d817-9f935c8c7acb","execution":{"iopub.status.busy":"2023-08-25T15:55:37.014646Z","iopub.execute_input":"2023-08-25T15:55:37.014975Z","iopub.status.idle":"2023-08-25T15:55:37.039553Z","shell.execute_reply.started":"2023-08-25T15:55:37.014938Z","shell.execute_reply":"2023-08-25T15:55:37.038408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"raw_df.target.value_counts(normalize=True).plot(kind=\"bar\")","metadata":{"executionInfo":{"elapsed":27,"status":"ok","timestamp":1692965493179,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"-nFN5bZVUL91","outputId":"c92d00fc-65d9-44b3-a404-8ef369287adb","execution":{"iopub.status.busy":"2023-08-25T15:55:37.044225Z","iopub.execute_input":"2023-08-25T15:55:37.044565Z","iopub.status.idle":"2023-08-25T15:55:37.347484Z","shell.execute_reply.started":"2023-08-25T15:55:37.044539Z","shell.execute_reply":"2023-08-25T15:55:37.346152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = pd.read_csv(test_fname)\ntest_df","metadata":{"executionInfo":{"elapsed":1619,"status":"ok","timestamp":1692965494778,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"n2VQSfKmUn7Z","outputId":"34698c21-2327-45f0-e4db-3d61dac91c8b","execution":{"iopub.status.busy":"2023-08-25T15:55:37.349248Z","iopub.execute_input":"2023-08-25T15:55:37.349973Z","iopub.status.idle":"2023-08-25T15:55:38.800347Z","shell.execute_reply.started":"2023-08-25T15:55:37.349927Z","shell.execute_reply":"2023-08-25T15:55:38.799242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_df = pd.read_csv(sample_fname)\nsample_df","metadata":{"executionInfo":{"elapsed":1064,"status":"ok","timestamp":1692965495833,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"ZSzHylWuU0MF","outputId":"654e4346-1c05-4e68-e571-33ddd3b3624d","execution":{"iopub.status.busy":"2023-08-25T15:55:38.801588Z","iopub.execute_input":"2023-08-25T15:55:38.801932Z","iopub.status.idle":"2023-08-25T15:55:39.185880Z","shell.execute_reply.started":"2023-08-25T15:55:38.801903Z","shell.execute_reply":"2023-08-25T15:55:39.184631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_df.prediction.value_counts()","metadata":{"executionInfo":{"elapsed":46,"status":"ok","timestamp":1692965495834,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"_lA6Vb6YVN1p","outputId":"56fd55b7-d938-4eae-f3ff-b0400f007426","execution":{"iopub.status.busy":"2023-08-25T15:55:39.187163Z","iopub.execute_input":"2023-08-25T15:55:39.187493Z","iopub.status.idle":"2023-08-25T15:55:39.200803Z","shell.execute_reply.started":"2023-08-25T15:55:39.187464Z","shell.execute_reply":"2023-08-25T15:55:39.199683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":37,"status":"ok","timestamp":1692965495835,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"gH0ij_pmVNzV"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":38,"status":"ok","timestamp":1692965495836,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"exDbcT9bVNw0"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":39,"status":"ok","timestamp":1692965495838,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"vLCvMxN3UL7l"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":40,"status":"ok","timestamp":1692965495839,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"BvL2NJuSVv4l"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create a Working Sample\n","metadata":{"id":"v7sWG0KCV0g-"}},{"cell_type":"code","source":"SAMPLE_SIZE = 100_000\nsample_df = raw_df.sample(SAMPLE_SIZE,random_state = 42)\nsample_df","metadata":{"executionInfo":{"elapsed":41,"status":"ok","timestamp":1692965495840,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"hu5s7hLKV7Kr","outputId":"10a5ac05-f7ac-4133-a6a7-ec48af07deba","execution":{"iopub.status.busy":"2023-08-25T15:55:39.202290Z","iopub.execute_input":"2023-08-25T15:55:39.202769Z","iopub.status.idle":"2023-08-25T15:55:39.303905Z","shell.execute_reply.started":"2023-08-25T15:55:39.202655Z","shell.execute_reply":"2023-08-25T15:55:39.302847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":40,"status":"ok","timestamp":1692965495841,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"0xzqyYo_V31J"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":40,"status":"ok","timestamp":1692965495842,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"4xFSOIfdV3yh"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Text Preprocessing Techniques\n\nOutline:\n\n1. Understand the bag of words model\n2. Tokenization\n3. Stop word removal\n4. Stemming","metadata":{"id":"-gZy068pQoPg"}},{"cell_type":"markdown","source":"### Bag of Words Intuition\n\n1. Create a list of all the words across all the text documents\n2. You convert each document into vector counts of each word\n\n\nLimitations:\n1. There may be too many words in the dataset\n2. Some words may occur too frequently\n3. Some words may occur very rarely or only once\n4. A single word may have many forms (go, gone, going or bird vs. birds)","metadata":{"id":"pJ0mh3gjXH9e"}},{"cell_type":"code","source":"q0 = sincere_df.question_text.values[1]\nq0","metadata":{"executionInfo":{"elapsed":39,"status":"ok","timestamp":1692965495842,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"PDb6D4pfYWPu","outputId":"a49b5c68-c6d0-4997-e5b1-dc3e4c67b6a5","execution":{"iopub.status.busy":"2023-08-25T15:55:39.305040Z","iopub.execute_input":"2023-08-25T15:55:39.305333Z","iopub.status.idle":"2023-08-25T15:55:39.314297Z","shell.execute_reply.started":"2023-08-25T15:55:39.305308Z","shell.execute_reply":"2023-08-25T15:55:39.313085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1 = insincere_df.question_text.values[4]\nq1","metadata":{"executionInfo":{"elapsed":36,"status":"ok","timestamp":1692965495843,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"mQjFxTKFYjJk","outputId":"f122e29c-f395-4284-c7f3-970837722d29","execution":{"iopub.status.busy":"2023-08-25T15:55:39.315219Z","iopub.execute_input":"2023-08-25T15:55:39.315522Z","iopub.status.idle":"2023-08-25T15:55:39.334310Z","shell.execute_reply.started":"2023-08-25T15:55:39.315495Z","shell.execute_reply":"2023-08-25T15:55:39.333174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":35,"status":"ok","timestamp":1692965495844,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"GIvt_M-yYlJH"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install nltk","metadata":{"executionInfo":{"elapsed":36,"status":"ok","timestamp":1692965495845,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"86bHxTMbY182","execution":{"iopub.status.busy":"2023-08-25T15:55:39.335374Z","iopub.execute_input":"2023-08-25T15:55:39.336825Z","iopub.status.idle":"2023-08-25T15:55:52.194710Z","shell.execute_reply.started":"2023-08-25T15:55:39.336790Z","shell.execute_reply":"2023-08-25T15:55:52.193701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Tokenization\n\nsplitting a document into words and separators","metadata":{"id":"qj4Zg1yaYPJv"}},{"cell_type":"code","source":"import nltk","metadata":{"executionInfo":{"elapsed":1979,"status":"ok","timestamp":1692965497788,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"EqacxiuyZPds","execution":{"iopub.status.busy":"2023-08-25T15:55:52.196522Z","iopub.execute_input":"2023-08-25T15:55:52.197006Z","iopub.status.idle":"2023-08-25T15:55:53.453103Z","shell.execute_reply.started":"2023-08-25T15:55:52.196946Z","shell.execute_reply":"2023-08-25T15:55:53.452259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from nltk.tokenize import word_tokenize","metadata":{"executionInfo":{"elapsed":107,"status":"ok","timestamp":1692965497789,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"2vQDMEq0Q_lP","execution":{"iopub.status.busy":"2023-08-25T15:55:53.455106Z","iopub.execute_input":"2023-08-25T15:55:53.455554Z","iopub.status.idle":"2023-08-25T15:55:53.460486Z","shell.execute_reply.started":"2023-08-25T15:55:53.455513Z","shell.execute_reply":"2023-08-25T15:55:53.459325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nltk.download('punkt')","metadata":{"executionInfo":{"elapsed":107,"status":"ok","timestamp":1692965497790,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"5eaAldKAZN-4","outputId":"d43a0b54-5c84-4be8-a82f-732b1288a73e","execution":{"iopub.status.busy":"2023-08-25T15:55:53.462156Z","iopub.execute_input":"2023-08-25T15:55:53.462772Z","iopub.status.idle":"2023-08-25T15:55:53.658303Z","shell.execute_reply.started":"2023-08-25T15:55:53.462739Z","shell.execute_reply":"2023-08-25T15:55:53.657261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"word_tokenize(q0)","metadata":{"executionInfo":{"elapsed":102,"status":"ok","timestamp":1692965497791,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"VyIaEATyQ_ip","outputId":"36517d88-07cf-4aa6-b6b8-82d8788a57e2","execution":{"iopub.status.busy":"2023-08-25T15:55:53.659928Z","iopub.execute_input":"2023-08-25T15:55:53.660565Z","iopub.status.idle":"2023-08-25T15:55:53.679757Z","shell.execute_reply.started":"2023-08-25T15:55:53.660523Z","shell.execute_reply":"2023-08-25T15:55:53.678648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":96,"status":"ok","timestamp":1692965497791,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"1yULG5nVQ_gk"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":96,"status":"ok","timestamp":1692965497792,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"gJ_lrgEpZIt8"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":97,"status":"ok","timestamp":1692965497793,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"uoP8H4KcZIr1"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":97,"status":"ok","timestamp":1692965497794,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"Bs_0YAawZIpf"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":97,"status":"ok","timestamp":1692965497794,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"7HNGqFYtZInd"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Stop Word Removal\n\nRemoving commonly occuring words","metadata":{"id":"Mf5aqvkGZrR-"}},{"cell_type":"code","source":"from nltk.corpus import stopwords\nnltk.download('stopwords')","metadata":{"executionInfo":{"elapsed":97,"status":"ok","timestamp":1692965497794,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"7yZhlXV9Zxo-","outputId":"75a29119-fd93-49a9-cd75-4fa9eb8765ed","execution":{"iopub.status.busy":"2023-08-25T15:55:53.681213Z","iopub.execute_input":"2023-08-25T15:55:53.681891Z","iopub.status.idle":"2023-08-25T15:55:53.694843Z","shell.execute_reply.started":"2023-08-25T15:55:53.681849Z","shell.execute_reply":"2023-08-25T15:55:53.693647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"english_stopwords = stopwords.words('english')","metadata":{"executionInfo":{"elapsed":93,"status":"ok","timestamp":1692965497795,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"s1qsENBhZrD0","execution":{"iopub.status.busy":"2023-08-25T15:55:53.696248Z","iopub.execute_input":"2023-08-25T15:55:53.696584Z","iopub.status.idle":"2023-08-25T15:55:53.703885Z","shell.execute_reply.started":"2023-08-25T15:55:53.696556Z","shell.execute_reply":"2023-08-25T15:55:53.703002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\", \".join(english_stopwords)","metadata":{"executionInfo":{"elapsed":92,"status":"ok","timestamp":1692965497795,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"4kjpGIjraK7r","outputId":"46860db9-6a2b-4b85-fa45-51b2be880db8","execution":{"iopub.status.busy":"2023-08-25T15:55:53.705192Z","iopub.execute_input":"2023-08-25T15:55:53.705932Z","iopub.status.idle":"2023-08-25T15:55:53.714905Z","shell.execute_reply.started":"2023-08-25T15:55:53.705899Z","shell.execute_reply":"2023-08-25T15:55:53.713834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def remove_stopwords(tokens):\n  return [word for word in tokens if word.lower() not in english_stopwords]","metadata":{"executionInfo":{"elapsed":90,"status":"ok","timestamp":1692965497796,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"ejhkI9oVZrCD","execution":{"iopub.status.busy":"2023-08-25T15:55:53.716509Z","iopub.execute_input":"2023-08-25T15:55:53.717065Z","iopub.status.idle":"2023-08-25T15:55:53.724518Z","shell.execute_reply.started":"2023-08-25T15:55:53.717025Z","shell.execute_reply":"2023-08-25T15:55:53.723741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0","metadata":{"executionInfo":{"elapsed":89,"status":"ok","timestamp":1692965497796,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"jB5avxBsZqxB","outputId":"58f820a7-d660-4efe-f153-5bc33adafe6e","execution":{"iopub.status.busy":"2023-08-25T15:55:53.728266Z","iopub.execute_input":"2023-08-25T15:55:53.728613Z","iopub.status.idle":"2023-08-25T15:55:53.740077Z","shell.execute_reply.started":"2023-08-25T15:55:53.728581Z","shell.execute_reply":"2023-08-25T15:55:53.738756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_tok = word_tokenize(q0)","metadata":{"executionInfo":{"elapsed":89,"status":"ok","timestamp":1692965497797,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"7umWfNCvZqsF","execution":{"iopub.status.busy":"2023-08-25T15:55:53.741957Z","iopub.execute_input":"2023-08-25T15:55:53.742340Z","iopub.status.idle":"2023-08-25T15:55:53.749367Z","shell.execute_reply.started":"2023-08-25T15:55:53.742305Z","shell.execute_reply":"2023-08-25T15:55:53.748465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_tok","metadata":{"executionInfo":{"elapsed":89,"status":"ok","timestamp":1692965497797,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"15NSoE_Saq6W","outputId":"3e28c07b-fe7e-4dc6-8b85-3c8d9a06513a","execution":{"iopub.status.busy":"2023-08-25T15:55:53.750696Z","iopub.execute_input":"2023-08-25T15:55:53.751701Z","iopub.status.idle":"2023-08-25T15:55:53.764558Z","shell.execute_reply.started":"2023-08-25T15:55:53.751645Z","shell.execute_reply":"2023-08-25T15:55:53.763648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_stp = remove_stopwords(q0_tok)\nq0_stp","metadata":{"executionInfo":{"elapsed":84,"status":"ok","timestamp":1692965497798,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"b59BPAClaq4B","outputId":"8dc9454e-2954-4af2-a0e4-ad2319399291","execution":{"iopub.status.busy":"2023-08-25T15:55:53.766130Z","iopub.execute_input":"2023-08-25T15:55:53.767023Z","iopub.status.idle":"2023-08-25T15:55:53.780218Z","shell.execute_reply.started":"2023-08-25T15:55:53.766988Z","shell.execute_reply":"2023-08-25T15:55:53.779349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_tok = word_tokenize(q1)\nq1_tok","metadata":{"executionInfo":{"elapsed":79,"status":"ok","timestamp":1692965497798,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"UJpOKfd-Zqp_","outputId":"fc56acf0-de08-49c1-b82d-dfad780dc8b0","execution":{"iopub.status.busy":"2023-08-25T15:55:53.781563Z","iopub.execute_input":"2023-08-25T15:55:53.782117Z","iopub.status.idle":"2023-08-25T15:55:53.793324Z","shell.execute_reply.started":"2023-08-25T15:55:53.782077Z","shell.execute_reply":"2023-08-25T15:55:53.792261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_stp = remove_stopwords(q1_tok)","metadata":{"executionInfo":{"elapsed":74,"status":"ok","timestamp":1692965497799,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"IKR9dyGVa6OR","execution":{"iopub.status.busy":"2023-08-25T15:55:53.802876Z","iopub.execute_input":"2023-08-25T15:55:53.803254Z","iopub.status.idle":"2023-08-25T15:55:53.807331Z","shell.execute_reply.started":"2023-08-25T15:55:53.803225Z","shell.execute_reply":"2023-08-25T15:55:53.806335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_stp","metadata":{"executionInfo":{"elapsed":74,"status":"ok","timestamp":1692965497799,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"_RvMiZAka6Lv","outputId":"afe33fa7-d1de-4a10-f965-423704af8614","execution":{"iopub.status.busy":"2023-08-25T15:55:53.808567Z","iopub.execute_input":"2023-08-25T15:55:53.809392Z","iopub.status.idle":"2023-08-25T15:55:53.821563Z","shell.execute_reply.started":"2023-08-25T15:55:53.809351Z","shell.execute_reply":"2023-08-25T15:55:53.820642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":70,"status":"ok","timestamp":1692965497800,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"GqpqLKw9a-A8"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Stemming\n\n> \"go\", \"gone\", \"going\" -> \"go\"\n>\n> \"birds\", \"bird\" -> \"bird\"","metadata":{"id":"FeVx68x5bHEF"}},{"cell_type":"code","source":"from nltk.stem.snowball import SnowballStemmer","metadata":{"executionInfo":{"elapsed":69,"status":"ok","timestamp":1692965497800,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"9jFOfSqtbG6y","execution":{"iopub.status.busy":"2023-08-25T15:55:53.822814Z","iopub.execute_input":"2023-08-25T15:55:53.823316Z","iopub.status.idle":"2023-08-25T15:55:53.832766Z","shell.execute_reply.started":"2023-08-25T15:55:53.823286Z","shell.execute_reply":"2023-08-25T15:55:53.831651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stemmer = SnowballStemmer(language='english')","metadata":{"executionInfo":{"elapsed":70,"status":"ok","timestamp":1692965497801,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"SC0PMIy3bG4r","execution":{"iopub.status.busy":"2023-08-25T15:55:53.834370Z","iopub.execute_input":"2023-08-25T15:55:53.835163Z","iopub.status.idle":"2023-08-25T15:55:53.843325Z","shell.execute_reply.started":"2023-08-25T15:55:53.835131Z","shell.execute_reply":"2023-08-25T15:55:53.842264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stemmer.stem('going')","metadata":{"executionInfo":{"elapsed":71,"status":"ok","timestamp":1692965497802,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"apl0nhFqbG2F","outputId":"5cd4eb14-8619-4073-be1b-80e7e180804b","execution":{"iopub.status.busy":"2023-08-25T15:55:53.844525Z","iopub.execute_input":"2023-08-25T15:55:53.844874Z","iopub.status.idle":"2023-08-25T15:55:53.860579Z","shell.execute_reply.started":"2023-08-25T15:55:53.844843Z","shell.execute_reply":"2023-08-25T15:55:53.859828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stemmer.stem('supposedly')","metadata":{"executionInfo":{"elapsed":71,"status":"ok","timestamp":1692965497804,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"wtIoHT1ybGz2","outputId":"3d3166b0-67be-48aa-940b-00eb7b8ec7d0","execution":{"iopub.status.busy":"2023-08-25T15:55:53.861975Z","iopub.execute_input":"2023-08-25T15:55:53.862391Z","iopub.status.idle":"2023-08-25T15:55:53.874150Z","shell.execute_reply.started":"2023-08-25T15:55:53.862351Z","shell.execute_reply":"2023-08-25T15:55:53.872886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_stm = [stemmer.stem(word) for word in q0_stp]\nq0_stm","metadata":{"executionInfo":{"elapsed":72,"status":"ok","timestamp":1692965497806,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"hGdECu-rbmbZ","outputId":"423894a0-65ed-4983-d4cf-e4c0954e048d","execution":{"iopub.status.busy":"2023-08-25T15:55:53.875760Z","iopub.execute_input":"2023-08-25T15:55:53.876176Z","iopub.status.idle":"2023-08-25T15:55:53.886491Z","shell.execute_reply.started":"2023-08-25T15:55:53.876138Z","shell.execute_reply":"2023-08-25T15:55:53.885783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_stm = [stemmer.stem(word) for word in q1_stp]\nq1_stm","metadata":{"executionInfo":{"elapsed":67,"status":"ok","timestamp":1692965497808,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"BY0fs95pbqtH","outputId":"245dd892-1f0c-4818-cada-0404b178ab82","execution":{"iopub.status.busy":"2023-08-25T15:55:53.887808Z","iopub.execute_input":"2023-08-25T15:55:53.888336Z","iopub.status.idle":"2023-08-25T15:55:53.899999Z","shell.execute_reply.started":"2023-08-25T15:55:53.888305Z","shell.execute_reply":"2023-08-25T15:55:53.898911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":64,"status":"ok","timestamp":1692965497810,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"ARuFpGkHbqn2"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":65,"status":"ok","timestamp":1692965497811,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"kiFiGdCpbrot"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":66,"status":"ok","timestamp":1692965497812,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"WweHQFO4brmn"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":66,"status":"ok","timestamp":1692965497813,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"AVu5FiAbbrkV"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Lemmatization\n\n\"love\" -> \"love\"\n>\n\"loving\" -> \"love\"\n>\n\"lovable\" -> \"love\"\n","metadata":{"id":"VWWkiTcWcAR7"}},{"cell_type":"markdown","source":"## Implement Bag of Words\n\n\nOutline:\n\n1. Create a vocabulary using Count Vectorizer\n2. Transform text to vectors using Count Vectorizer\n3. Configure text preprocessing in Count Vectorizer","metadata":{"id":"CuTR9sOWQoMq"}},{"cell_type":"markdown","source":"### Create a Vocabulary","metadata":{"id":"0HWShc6jdPFy"}},{"cell_type":"code","source":"sample_df.shape","metadata":{"executionInfo":{"elapsed":67,"status":"ok","timestamp":1692965497814,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"viRMtlD8mja6","outputId":"a1706d7b-cf3b-4b86-ac30-a52c85802e57","execution":{"iopub.status.busy":"2023-08-25T15:55:53.901356Z","iopub.execute_input":"2023-08-25T15:55:53.901901Z","iopub.status.idle":"2023-08-25T15:55:53.912253Z","shell.execute_reply.started":"2023-08-25T15:55:53.901870Z","shell.execute_reply":"2023-08-25T15:55:53.911209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_df = sample_df[:5]","metadata":{"executionInfo":{"elapsed":1148,"status":"ok","timestamp":1692965781212,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"uGScEAd9c_dQ","execution":{"iopub.status.busy":"2023-08-25T15:55:53.913196Z","iopub.execute_input":"2023-08-25T15:55:53.913482Z","iopub.status.idle":"2023-08-25T15:55:53.922170Z","shell.execute_reply.started":"2023-08-25T15:55:53.913457Z","shell.execute_reply":"2023-08-25T15:55:53.921191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_df.shape","metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1692965783657,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"IKj0BM2NRATm","outputId":"afe21adf-246a-4925-be33-1a82072169b4","execution":{"iopub.status.busy":"2023-08-25T15:55:53.923747Z","iopub.execute_input":"2023-08-25T15:55:53.924150Z","iopub.status.idle":"2023-08-25T15:55:53.934483Z","shell.execute_reply.started":"2023-08-25T15:55:53.924112Z","shell.execute_reply":"2023-08-25T15:55:53.933411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer","metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1692965787141,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"awqkIOZEdKFu","execution":{"iopub.status.busy":"2023-08-25T15:55:53.936089Z","iopub.execute_input":"2023-08-25T15:55:53.936492Z","iopub.status.idle":"2023-08-25T15:55:53.944499Z","shell.execute_reply.started":"2023-08-25T15:55:53.936454Z","shell.execute_reply":"2023-08-25T15:55:53.943728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_vect = CountVectorizer()","metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1692965788089,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"XZOIXmC3RARh","execution":{"iopub.status.busy":"2023-08-25T15:55:53.945975Z","iopub.execute_input":"2023-08-25T15:55:53.946364Z","iopub.status.idle":"2023-08-25T15:55:53.955376Z","shell.execute_reply.started":"2023-08-25T15:55:53.946328Z","shell.execute_reply":"2023-08-25T15:55:53.954443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_vect.fit(small_df.question_text)","metadata":{"executionInfo":{"elapsed":10,"status":"ok","timestamp":1692965789290,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"Yr3XgzBWdNNx","outputId":"e264b8a4-2624-46e5-9f6f-a77611b13997","execution":{"iopub.status.busy":"2023-08-25T15:55:53.956921Z","iopub.execute_input":"2023-08-25T15:55:53.957328Z","iopub.status.idle":"2023-08-25T15:55:53.981501Z","shell.execute_reply.started":"2023-08-25T15:55:53.957291Z","shell.execute_reply":"2023-08-25T15:55:53.980515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_vect.vocabulary_","metadata":{"executionInfo":{"elapsed":9,"status":"ok","timestamp":1692965789291,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"-sE8WL3-dNLf","outputId":"5d493da2-b6cf-4d0a-dec4-6ad8482dafa9","execution":{"iopub.status.busy":"2023-08-25T15:55:53.982925Z","iopub.execute_input":"2023-08-25T15:55:53.983454Z","iopub.status.idle":"2023-08-25T15:55:53.991031Z","shell.execute_reply.started":"2023-08-25T15:55:53.983421Z","shell.execute_reply":"2023-08-25T15:55:53.990191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_vect.get_feature_names_out()","metadata":{"executionInfo":{"elapsed":626,"status":"ok","timestamp":1692965837544,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"ba9oOkD6RAPP","outputId":"262d87a9-5a98-47cc-be80-a33b81a8cb25","execution":{"iopub.status.busy":"2023-08-25T15:55:53.992070Z","iopub.execute_input":"2023-08-25T15:55:53.992839Z","iopub.status.idle":"2023-08-25T15:55:54.005649Z","shell.execute_reply.started":"2023-08-25T15:55:53.992808Z","shell.execute_reply":"2023-08-25T15:55:54.004834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":69,"status":"ok","timestamp":1692965497824,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"-QPGkxLZdkPy"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Transform documents into Vectors","metadata":{"id":"WXsUWz_hd9KE"}},{"cell_type":"code","source":"vectors = small_vect.transform(small_df.question_text.values)","metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1692965954751,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"nZVZ3jIddkKT","execution":{"iopub.status.busy":"2023-08-25T15:55:54.006654Z","iopub.execute_input":"2023-08-25T15:55:54.007450Z","iopub.status.idle":"2023-08-25T15:55:54.014877Z","shell.execute_reply.started":"2023-08-25T15:55:54.007420Z","shell.execute_reply":"2023-08-25T15:55:54.013917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectors","metadata":{"executionInfo":{"elapsed":7,"status":"ok","timestamp":1692965958976,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"MIGbsQEQdkHt","outputId":"2063d017-7762-4deb-ad45-c02af8e6f444","execution":{"iopub.status.busy":"2023-08-25T15:55:54.016198Z","iopub.execute_input":"2023-08-25T15:55:54.016833Z","iopub.status.idle":"2023-08-25T15:55:54.027425Z","shell.execute_reply.started":"2023-08-25T15:55:54.016802Z","shell.execute_reply":"2023-08-25T15:55:54.026680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectors.toarray()","metadata":{"executionInfo":{"elapsed":6,"status":"ok","timestamp":1692966006024,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"wriwC-1DeZPA","outputId":"9a51c004-3da8-4933-c722-c66857b84d2b","execution":{"iopub.status.busy":"2023-08-25T15:55:54.028505Z","iopub.execute_input":"2023-08-25T15:55:54.028960Z","iopub.status.idle":"2023-08-25T15:55:54.044260Z","shell.execute_reply.started":"2023-08-25T15:55:54.028933Z","shell.execute_reply":"2023-08-25T15:55:54.043045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectors.shape","metadata":{"executionInfo":{"elapsed":7,"status":"ok","timestamp":1692966011297,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"bXtFEWjtekqt","outputId":"1387eb21-88d7-48e5-b42a-28a526df0815","execution":{"iopub.status.busy":"2023-08-25T15:55:54.045808Z","iopub.execute_input":"2023-08-25T15:55:54.046548Z","iopub.status.idle":"2023-08-25T15:55:54.055658Z","shell.execute_reply.started":"2023-08-25T15:55:54.046507Z","shell.execute_reply":"2023-08-25T15:55:54.054921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectors[0].toarray()","metadata":{"executionInfo":{"elapsed":6,"status":"ok","timestamp":1692966234445,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"etQ-T7_1e0vF","outputId":"b2890929-0ec5-44b8-85a9-09995511d1b7","execution":{"iopub.status.busy":"2023-08-25T15:55:54.056820Z","iopub.execute_input":"2023-08-25T15:55:54.057380Z","iopub.status.idle":"2023-08-25T15:55:54.068476Z","shell.execute_reply.started":"2023-08-25T15:55:54.057349Z","shell.execute_reply":"2023-08-25T15:55:54.067731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_df.question_text.values[0]","metadata":{"executionInfo":{"elapsed":617,"status":"ok","timestamp":1692966277560,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"hSrf1ZPndkFX","outputId":"122fe0b6-02b7-4b61-a3cd-75955d0fdea0","execution":{"iopub.status.busy":"2023-08-25T15:55:54.069782Z","iopub.execute_input":"2023-08-25T15:55:54.070585Z","iopub.status.idle":"2023-08-25T15:55:54.082455Z","shell.execute_reply.started":"2023-08-25T15:55:54.070555Z","shell.execute_reply":"2023-08-25T15:55:54.081596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Configure Count Vectorizer Parameters","metadata":{"id":"vZPCOwobe485"}},{"cell_type":"code","source":"stemmer = SnowballStemmer(language = 'english')","metadata":{"id":"Gx4zqV4Grzg6","execution":{"iopub.status.busy":"2023-08-25T15:55:54.083714Z","iopub.execute_input":"2023-08-25T15:55:54.084283Z","iopub.status.idle":"2023-08-25T15:55:54.095756Z","shell.execute_reply.started":"2023-08-25T15:55:54.084252Z","shell.execute_reply":"2023-08-25T15:55:54.094872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def tokenize(text):\n  return [stemmer.stem(word) for word in word_tokenize(text) if word.lower() not in english_stopwords]","metadata":{"executionInfo":{"elapsed":4,"status":"ok","timestamp":1692966836276,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"9hZzn1LBfPoM","execution":{"iopub.status.busy":"2023-08-25T15:55:54.096947Z","iopub.execute_input":"2023-08-25T15:55:54.097740Z","iopub.status.idle":"2023-08-25T15:55:54.109946Z","shell.execute_reply.started":"2023-08-25T15:55:54.097695Z","shell.execute_reply":"2023-08-25T15:55:54.109094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tokenize(\"what is the (real) dealing <here>\")","metadata":{"executionInfo":{"elapsed":7,"status":"ok","timestamp":1692966871604,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"OAausoIrfa2I","outputId":"f05b714c-2996-4fcc-acbf-80988fe99b66","execution":{"iopub.status.busy":"2023-08-25T15:55:54.111265Z","iopub.execute_input":"2023-08-25T15:55:54.111823Z","iopub.status.idle":"2023-08-25T15:55:54.125791Z","shell.execute_reply.started":"2023-08-25T15:55:54.111786Z","shell.execute_reply":"2023-08-25T15:55:54.124739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectorizer = CountVectorizer(\n    lowercase = True,\n    tokenizer = tokenize,\n    stop_words = english_stopwords,\n    max_features = 1000\n)","metadata":{"executionInfo":{"elapsed":9,"status":"ok","timestamp":1692966968993,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"xUjZDsdqfq4u","execution":{"iopub.status.busy":"2023-08-25T15:55:54.127039Z","iopub.execute_input":"2023-08-25T15:55:54.127344Z","iopub.status.idle":"2023-08-25T15:55:54.135602Z","shell.execute_reply.started":"2023-08-25T15:55:54.127317Z","shell.execute_reply":"2023-08-25T15:55:54.134713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nvectorizer.fit(sample_df.question_text)","metadata":{"executionInfo":{"elapsed":29528,"status":"ok","timestamp":1692967056337,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"pJGAErs7e9Fh","outputId":"8d948769-2870-4eda-a557-0ff87ef7f3bc","execution":{"iopub.status.busy":"2023-08-25T15:55:54.136580Z","iopub.execute_input":"2023-08-25T15:55:54.137057Z","iopub.status.idle":"2023-08-25T15:56:25.462366Z","shell.execute_reply.started":"2023-08-25T15:55:54.137028Z","shell.execute_reply":"2023-08-25T15:56:25.461336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(vectorizer.vocabulary_)","metadata":{"executionInfo":{"elapsed":7,"status":"ok","timestamp":1692967064535,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"pI8nbkpAe9Dg","outputId":"2adac301-548c-4f03-919a-48584a7f99ca","execution":{"iopub.status.busy":"2023-08-25T15:56:25.463911Z","iopub.execute_input":"2023-08-25T15:56:25.464260Z","iopub.status.idle":"2023-08-25T15:56:25.471596Z","shell.execute_reply.started":"2023-08-25T15:56:25.464228Z","shell.execute_reply":"2023-08-25T15:56:25.470412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectorizer.get_feature_names_out()[:40]","metadata":{"executionInfo":{"elapsed":8,"status":"ok","timestamp":1692967092199,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"sJBlLm5Ye88Z","outputId":"7efad3b4-a65f-4bd9-ce69-c847c1f44bad","execution":{"iopub.status.busy":"2023-08-25T15:56:25.473556Z","iopub.execute_input":"2023-08-25T15:56:25.474066Z","iopub.status.idle":"2023-08-25T15:56:25.486917Z","shell.execute_reply.started":"2023-08-25T15:56:25.474030Z","shell.execute_reply":"2023-08-25T15:56:25.485608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ninputs = vectorizer.transform(sample_df.question_text)","metadata":{"executionInfo":{"elapsed":28794,"status":"ok","timestamp":1692967223685,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"b9K638h1gg5_","outputId":"bb41cab7-87ed-4c1c-aeba-4cf51ab72ebe","execution":{"iopub.status.busy":"2023-08-25T15:56:25.488403Z","iopub.execute_input":"2023-08-25T15:56:25.488868Z","iopub.status.idle":"2023-08-25T15:56:56.539754Z","shell.execute_reply.started":"2023-08-25T15:56:25.488825Z","shell.execute_reply":"2023-08-25T15:56:56.538983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"inputs[0].toarray()","metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1692967307244,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"n3iq65OsguTR","outputId":"31d9bf8f-c882-46b1-e848-d0d47e4d6b16","execution":{"iopub.status.busy":"2023-08-25T15:56:56.541329Z","iopub.execute_input":"2023-08-25T15:56:56.542050Z","iopub.status.idle":"2023-08-25T15:56:56.552617Z","shell.execute_reply.started":"2023-08-25T15:56:56.542006Z","shell.execute_reply":"2023-08-25T15:56:56.551313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.shape","metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1692967909312,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"9aEbKIC0gg3o","outputId":"f2446fca-17c7-4132-cee7-1372c96e6c67","execution":{"iopub.status.busy":"2023-08-25T15:56:56.553943Z","iopub.execute_input":"2023-08-25T15:56:56.554298Z","iopub.status.idle":"2023-08-25T15:56:56.564217Z","shell.execute_reply.started":"2023-08-25T15:56:56.554265Z","shell.execute_reply":"2023-08-25T15:56:56.563209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntest_inputs = vectorizer.transform(test_df.question_text)","metadata":{"executionInfo":{"elapsed":96404,"status":"ok","timestamp":1692968036385,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"meslQK7Le86I","outputId":"a3a8d05b-ae8f-494a-d19f-cb88ebab8fb9","execution":{"iopub.status.busy":"2023-08-25T15:56:56.566115Z","iopub.execute_input":"2023-08-25T15:56:56.566789Z","iopub.status.idle":"2023-08-25T15:58:51.893898Z","shell.execute_reply.started":"2023-08-25T15:56:56.566729Z","shell.execute_reply":"2023-08-25T15:58:51.892823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":36,"status":"ok","timestamp":1692965499019,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"ARTJ68oehHVR"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":36,"status":"ok","timestamp":1692965499019,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"fJ497jJyhCaK"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":37,"status":"ok","timestamp":1692965499020,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"UDu0wUf9g--d"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":38,"status":"ok","timestamp":1692965499022,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"CKLhsg9GdkDW"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ML Models for Text Classification\n\nOutline:\n\n- Create a training & validation set\n- Train a logistic regression model\n- Make predictions on training, validation & test data","metadata":{"id":"AyVwmOboQoJn"}},{"cell_type":"markdown","source":"### Split into Training and Validation Set","metadata":{"id":"1CPsktK-iPgK"}},{"cell_type":"code","source":"sample_df.shape","metadata":{"executionInfo":{"elapsed":8,"status":"ok","timestamp":1692968288983,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"hoIv721biUVy","outputId":"919984b6-2419-4cad-ed4c-fabf9fee9eb8","execution":{"iopub.status.busy":"2023-08-25T15:58:51.895243Z","iopub.execute_input":"2023-08-25T15:58:51.895951Z","iopub.status.idle":"2023-08-25T15:58:51.902638Z","shell.execute_reply.started":"2023-08-25T15:58:51.895912Z","shell.execute_reply":"2023-08-25T15:58:51.901753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"inputs.shape","metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1692968317042,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"ckPyKMcMRBET","outputId":"6fc063db-f056-405c-a5dc-b5a33b313b3b","execution":{"iopub.status.busy":"2023-08-25T15:58:51.904123Z","iopub.execute_input":"2023-08-25T15:58:51.904458Z","iopub.status.idle":"2023-08-25T15:58:51.918488Z","shell.execute_reply.started":"2023-08-25T15:58:51.904427Z","shell.execute_reply":"2023-08-25T15:58:51.917455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split","metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1692968399096,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"m3eE_6y6ieLP","execution":{"iopub.status.busy":"2023-08-25T15:58:51.919863Z","iopub.execute_input":"2023-08-25T15:58:51.920224Z","iopub.status.idle":"2023-08-25T15:58:51.928719Z","shell.execute_reply.started":"2023-08-25T15:58:51.920194Z","shell.execute_reply":"2023-08-25T15:58:51.927696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_inputs,val_inputs,train_targets,val_targets = train_test_split(inputs,sample_df.target,test_size=0.3,random_state = 42)","metadata":{"executionInfo":{"elapsed":1112,"status":"ok","timestamp":1692968487238,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"OB2jqu7dirOn","execution":{"iopub.status.busy":"2023-08-25T15:58:51.929996Z","iopub.execute_input":"2023-08-25T15:58:51.930302Z","iopub.status.idle":"2023-08-25T15:58:51.964017Z","shell.execute_reply.started":"2023-08-25T15:58:51.930277Z","shell.execute_reply":"2023-08-25T15:58:51.963101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_inputs.shape","metadata":{"executionInfo":{"elapsed":621,"status":"ok","timestamp":1692968499524,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"1hMoWlcrRBBt","outputId":"f05cf288-523a-43d9-c166-fa2400b1d056","execution":{"iopub.status.busy":"2023-08-25T15:58:51.965329Z","iopub.execute_input":"2023-08-25T15:58:51.966329Z","iopub.status.idle":"2023-08-25T15:58:51.972648Z","shell.execute_reply.started":"2023-08-25T15:58:51.966290Z","shell.execute_reply":"2023-08-25T15:58:51.971603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_targets.shape","metadata":{"executionInfo":{"elapsed":890,"status":"ok","timestamp":1692968513452,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"ioOd41xKjCkX","outputId":"a7359507-d2c7-49b3-aa44-f8a9540ef3f9","execution":{"iopub.status.busy":"2023-08-25T15:58:51.973872Z","iopub.execute_input":"2023-08-25T15:58:51.974186Z","iopub.status.idle":"2023-08-25T15:58:51.985362Z","shell.execute_reply.started":"2023-08-25T15:58:51.974158Z","shell.execute_reply":"2023-08-25T15:58:51.984134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_inputs.shape","metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1692968522982,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"d2ipJkxRjChw","outputId":"090fef0c-ce81-4ef2-e690-07c46a81feaa","execution":{"iopub.status.busy":"2023-08-25T15:58:51.986683Z","iopub.execute_input":"2023-08-25T15:58:51.987023Z","iopub.status.idle":"2023-08-25T15:58:51.998791Z","shell.execute_reply.started":"2023-08-25T15:58:51.986996Z","shell.execute_reply":"2023-08-25T15:58:51.997854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_targets.shape","metadata":{"executionInfo":{"elapsed":9,"status":"ok","timestamp":1692968530580,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"TZuuLryBjIWZ","outputId":"c5a63e8f-0643-42de-d7b3-8eb63042f341","execution":{"iopub.status.busy":"2023-08-25T15:58:52.000226Z","iopub.execute_input":"2023-08-25T15:58:52.000833Z","iopub.status.idle":"2023-08-25T15:58:52.012507Z","shell.execute_reply.started":"2023-08-25T15:58:52.000794Z","shell.execute_reply":"2023-08-25T15:58:52.011349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train Logistic Regression model","metadata":{"id":"bBAcDSenjMSk"}},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression","metadata":{"executionInfo":{"elapsed":8,"status":"ok","timestamp":1692968721857,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"tURg-O4pjMmX","execution":{"iopub.status.busy":"2023-08-25T15:58:52.014358Z","iopub.execute_input":"2023-08-25T15:58:52.015332Z","iopub.status.idle":"2023-08-25T15:58:52.022586Z","shell.execute_reply.started":"2023-08-25T15:58:52.015293Z","shell.execute_reply":"2023-08-25T15:58:52.021464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = LogisticRegression(max_iter=1000,solver='sag')","metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1692968916794,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"aqk9RFIYj9tu","execution":{"iopub.status.busy":"2023-08-25T15:58:52.023985Z","iopub.execute_input":"2023-08-25T15:58:52.024307Z","iopub.status.idle":"2023-08-25T15:58:52.035203Z","shell.execute_reply.started":"2023-08-25T15:58:52.024279Z","shell.execute_reply":"2023-08-25T15:58:52.034052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MAX_ITER = 1000","metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1692968916795,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"fOehu5rLjNFl","execution":{"iopub.status.busy":"2023-08-25T15:58:52.036718Z","iopub.execute_input":"2023-08-25T15:58:52.037800Z","iopub.status.idle":"2023-08-25T15:58:52.047813Z","shell.execute_reply.started":"2023-08-25T15:58:52.037764Z","shell.execute_reply":"2023-08-25T15:58:52.046658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nmodel.fit(train_inputs,train_targets)","metadata":{"executionInfo":{"elapsed":14249,"status":"ok","timestamp":1692968955027,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"uUQ0wzNIjNHr","outputId":"8175b9f3-f801-45a2-ae27-a0add61398ff","execution":{"iopub.status.busy":"2023-08-25T15:58:52.049506Z","iopub.execute_input":"2023-08-25T15:58:52.050526Z","iopub.status.idle":"2023-08-25T15:59:19.818574Z","shell.execute_reply.started":"2023-08-25T15:58:52.050491Z","shell.execute_reply":"2023-08-25T15:59:19.817545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Make predictions using the model","metadata":{"id":"o16-8HmXmQT0"}},{"cell_type":"code","source":"train_preds = model.predict(train_inputs)","metadata":{"executionInfo":{"elapsed":3,"status":"ok","timestamp":1692968982574,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"i12DBlkVkThU","execution":{"iopub.status.busy":"2023-08-25T15:59:19.819803Z","iopub.execute_input":"2023-08-25T15:59:19.820124Z","iopub.status.idle":"2023-08-25T15:59:19.829065Z","shell.execute_reply.started":"2023-08-25T15:59:19.820095Z","shell.execute_reply":"2023-08-25T15:59:19.828054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_preds","metadata":{"executionInfo":{"elapsed":8,"status":"ok","timestamp":1692968992606,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"urNCyppPkxZB","outputId":"a9bb8fa7-3ad8-45a7-b960-bf89b5b9c141","execution":{"iopub.status.busy":"2023-08-25T15:59:19.830460Z","iopub.execute_input":"2023-08-25T15:59:19.830812Z","iopub.status.idle":"2023-08-25T15:59:19.840348Z","shell.execute_reply.started":"2023-08-25T15:59:19.830782Z","shell.execute_reply":"2023-08-25T15:59:19.839431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series(train_preds).value_counts()","metadata":{"executionInfo":{"elapsed":6,"status":"ok","timestamp":1692969012202,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"qS8PDYxljNDP","outputId":"377524c7-693a-4b72-cf10-394f1e8e0ba0","execution":{"iopub.status.busy":"2023-08-25T15:59:19.842056Z","iopub.execute_input":"2023-08-25T15:59:19.842362Z","iopub.status.idle":"2023-08-25T15:59:19.857685Z","shell.execute_reply.started":"2023-08-25T15:59:19.842336Z","shell.execute_reply":"2023-08-25T15:59:19.856559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series(train_targets).value_counts()","metadata":{"executionInfo":{"elapsed":806,"status":"ok","timestamp":1692969035303,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"W74eTiJgjNA9","outputId":"68e50608-a232-4091-e2d6-ae911888ca1b","execution":{"iopub.status.busy":"2023-08-25T15:59:19.858842Z","iopub.execute_input":"2023-08-25T15:59:19.859146Z","iopub.status.idle":"2023-08-25T15:59:19.878200Z","shell.execute_reply.started":"2023-08-25T15:59:19.859120Z","shell.execute_reply":"2023-08-25T15:59:19.877362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import accuracy_score","metadata":{"executionInfo":{"elapsed":3,"status":"ok","timestamp":1692969071144,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"pixsvLxrkhK4","execution":{"iopub.status.busy":"2023-08-25T15:59:19.879241Z","iopub.execute_input":"2023-08-25T15:59:19.879981Z","iopub.status.idle":"2023-08-25T15:59:19.888349Z","shell.execute_reply.started":"2023-08-25T15:59:19.879950Z","shell.execute_reply":"2023-08-25T15:59:19.887263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_score(train_targets,train_preds)","metadata":{"executionInfo":{"elapsed":7,"status":"ok","timestamp":1692969087427,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"YxH33IpFkpXJ","outputId":"f016db01-7033-41a8-a044-7ced133c048f","execution":{"iopub.status.busy":"2023-08-25T15:59:19.889935Z","iopub.execute_input":"2023-08-25T15:59:19.890413Z","iopub.status.idle":"2023-08-25T15:59:19.909809Z","shell.execute_reply.started":"2023-08-25T15:59:19.890380Z","shell.execute_reply":"2023-08-25T15:59:19.908614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np","metadata":{"executionInfo":{"elapsed":561,"status":"ok","timestamp":1692969160534,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"ipzg1inClCcJ","execution":{"iopub.status.busy":"2023-08-25T15:59:19.911293Z","iopub.execute_input":"2023-08-25T15:59:19.912192Z","iopub.status.idle":"2023-08-25T15:59:19.916409Z","shell.execute_reply.started":"2023-08-25T15:59:19.912155Z","shell.execute_reply":"2023-08-25T15:59:19.915503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_score(train_targets,np.zeros(len(train_targets)))","metadata":{"executionInfo":{"elapsed":11,"status":"ok","timestamp":1692969160535,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"oUgKam9RkYD6","outputId":"94827422-6125-4902-b6d6-4f413d8b34e8","execution":{"iopub.status.busy":"2023-08-25T15:59:19.917783Z","iopub.execute_input":"2023-08-25T15:59:19.918134Z","iopub.status.idle":"2023-08-25T15:59:19.941837Z","shell.execute_reply.started":"2023-08-25T15:59:19.918094Z","shell.execute_reply":"2023-08-25T15:59:19.940386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import f1_score","metadata":{"executionInfo":{"elapsed":4,"status":"ok","timestamp":1692969199970,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"_P-IsOtTk5x_","execution":{"iopub.status.busy":"2023-08-25T15:59:19.943596Z","iopub.execute_input":"2023-08-25T15:59:19.944742Z","iopub.status.idle":"2023-08-25T15:59:19.949748Z","shell.execute_reply.started":"2023-08-25T15:59:19.944695Z","shell.execute_reply":"2023-08-25T15:59:19.948742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f1_score(train_targets,train_preds)","metadata":{"executionInfo":{"elapsed":626,"status":"ok","timestamp":1692969215240,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"89TtZhqclKwb","outputId":"a839b2bc-f129-4952-dd80-88ba8f05e7d7","execution":{"iopub.status.busy":"2023-08-25T15:59:19.951064Z","iopub.execute_input":"2023-08-25T15:59:19.951487Z","iopub.status.idle":"2023-08-25T15:59:19.989801Z","shell.execute_reply.started":"2023-08-25T15:59:19.951447Z","shell.execute_reply":"2023-08-25T15:59:19.988829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f1_score(train_targets,np.zeros(len(train_targets)))","metadata":{"executionInfo":{"elapsed":12,"status":"ok","timestamp":1692969251794,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"pim_sjENlKn0","outputId":"f20f92f9-ea0a-4ea0-e47b-0f474ab0376c","execution":{"iopub.status.busy":"2023-08-25T15:59:19.991614Z","iopub.execute_input":"2023-08-25T15:59:19.991943Z","iopub.status.idle":"2023-08-25T15:59:20.030119Z","shell.execute_reply.started":"2023-08-25T15:59:19.991916Z","shell.execute_reply":"2023-08-25T15:59:20.028998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f1_score(train_targets,np.random.choice((0,1),(len(train_targets))))","metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1692969317938,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"RembBU-Fksbk","outputId":"54995e3b-d12c-4f29-e27e-a0dd52c5cbcc","execution":{"iopub.status.busy":"2023-08-25T15:59:20.031487Z","iopub.execute_input":"2023-08-25T15:59:20.031937Z","iopub.status.idle":"2023-08-25T15:59:20.069726Z","shell.execute_reply.started":"2023-08-25T15:59:20.031898Z","shell.execute_reply":"2023-08-25T15:59:20.068716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_preds = model.predict(val_inputs)","metadata":{"executionInfo":{"elapsed":4,"status":"ok","timestamp":1692969403826,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"43hI8WB8ld0e","execution":{"iopub.status.busy":"2023-08-25T15:59:20.071153Z","iopub.execute_input":"2023-08-25T15:59:20.071830Z","iopub.status.idle":"2023-08-25T15:59:20.078806Z","shell.execute_reply.started":"2023-08-25T15:59:20.071789Z","shell.execute_reply":"2023-08-25T15:59:20.077595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_score(val_targets,val_preds)","metadata":{"executionInfo":{"elapsed":552,"status":"ok","timestamp":1692969430709,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"FM0X5ImAl7gB","outputId":"01d6cde4-8abc-4edc-e236-9572c57f3638","execution":{"iopub.status.busy":"2023-08-25T15:59:20.080031Z","iopub.execute_input":"2023-08-25T15:59:20.080623Z","iopub.status.idle":"2023-08-25T15:59:20.095043Z","shell.execute_reply.started":"2023-08-25T15:59:20.080589Z","shell.execute_reply":"2023-08-25T15:59:20.093997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f1_score(val_targets,val_preds)","metadata":{"executionInfo":{"elapsed":6,"status":"ok","timestamp":1692969447807,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"WiXz18lal7db","outputId":"0f8aa51c-07bf-4c5b-d707-5767abc7542c","execution":{"iopub.status.busy":"2023-08-25T15:59:20.096435Z","iopub.execute_input":"2023-08-25T15:59:20.097363Z","iopub.status.idle":"2023-08-25T15:59:20.117584Z","shell.execute_reply.started":"2023-08-25T15:59:20.097244Z","shell.execute_reply":"2023-08-25T15:59:20.116459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sincere_df.question_text.values[:10]","metadata":{"executionInfo":{"elapsed":6,"status":"ok","timestamp":1692969514969,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"Kj-oIhIHksY-","outputId":"c3ca11bb-cae5-4749-c4e4-d34d2ded73be","execution":{"iopub.status.busy":"2023-08-25T15:59:20.118906Z","iopub.execute_input":"2023-08-25T15:59:20.119203Z","iopub.status.idle":"2023-08-25T15:59:20.124737Z","shell.execute_reply.started":"2023-08-25T15:59:20.119178Z","shell.execute_reply":"2023-08-25T15:59:20.123952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sincere_df.target.values[:10]","metadata":{"executionInfo":{"elapsed":6,"status":"ok","timestamp":1692969554915,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"qxCSeXZQmHpS","outputId":"b65bcc0c-3abb-4488-dc6a-7c4938b2d1e4","execution":{"iopub.status.busy":"2023-08-25T15:59:20.126218Z","iopub.execute_input":"2023-08-25T15:59:20.126751Z","iopub.status.idle":"2023-08-25T15:59:20.138517Z","shell.execute_reply.started":"2023-08-25T15:59:20.126716Z","shell.execute_reply":"2023-08-25T15:59:20.137317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.predict(vectorizer.transform(sincere_df.question_text.values[:10]))","metadata":{"executionInfo":{"elapsed":9,"status":"ok","timestamp":1692969600993,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"fwlbPtLDmgDf","outputId":"676e56d9-44ca-4762-8d95-cc770fcfa1e8","execution":{"iopub.status.busy":"2023-08-25T15:59:20.139977Z","iopub.execute_input":"2023-08-25T15:59:20.140311Z","iopub.status.idle":"2023-08-25T15:59:20.154516Z","shell.execute_reply.started":"2023-08-25T15:59:20.140283Z","shell.execute_reply":"2023-08-25T15:59:20.153446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":41,"status":"ok","timestamp":1692965499035,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"qHE1gR0mmV0l"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":41,"status":"ok","timestamp":1692965499035,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"k58qFC-kmVyR"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":41,"status":"ok","timestamp":1692965499035,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"lFwis7sOmVv7"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":40,"status":"ok","timestamp":1692965499035,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"2T_zvDYvmVt6"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Make Predictions and Submit to Kaggle","metadata":{"id":"TSFA3r48Q2ec"}},{"cell_type":"code","source":"test_df","metadata":{"executionInfo":{"elapsed":11,"status":"ok","timestamp":1692969889996,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"bIAZhmcWQDd4","outputId":"287eb5e1-e6ce-4a23-e48a-26e237ce1bf0","execution":{"iopub.status.busy":"2023-08-25T15:59:20.165768Z","iopub.execute_input":"2023-08-25T15:59:20.166381Z","iopub.status.idle":"2023-08-25T15:59:20.178005Z","shell.execute_reply.started":"2023-08-25T15:59:20.166348Z","shell.execute_reply":"2023-08-25T15:59:20.176914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_inputs","metadata":{"executionInfo":{"elapsed":7,"status":"ok","timestamp":1692969906053,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"jzb8x69DnoHz","outputId":"4dbad93a-6719-41e2-f2fc-0838347beb69","execution":{"iopub.status.busy":"2023-08-25T15:59:20.179597Z","iopub.execute_input":"2023-08-25T15:59:20.179983Z","iopub.status.idle":"2023-08-25T15:59:20.191063Z","shell.execute_reply.started":"2023-08-25T15:59:20.179935Z","shell.execute_reply":"2023-08-25T15:59:20.190057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_preds = model.predict(test_inputs)","metadata":{"executionInfo":{"elapsed":4,"status":"ok","timestamp":1692969935032,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"oiY8Ai-anqRP","execution":{"iopub.status.busy":"2023-08-25T15:59:20.192394Z","iopub.execute_input":"2023-08-25T15:59:20.192834Z","iopub.status.idle":"2023-08-25T15:59:20.220201Z","shell.execute_reply.started":"2023-08-25T15:59:20.192800Z","shell.execute_reply":"2023-08-25T15:59:20.219210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# sub_df = pd.read_csv('/content/data/sample_submission.csv.zip')\nsub_df = pd.read_csv('/kaggle/input/quora-insincere-questions-classification/sample_submission.csv')","metadata":{"executionInfo":{"elapsed":924,"status":"ok","timestamp":1692970035622,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"W8h1xhbZnwBt","execution":{"iopub.status.busy":"2023-08-25T15:59:20.221412Z","iopub.execute_input":"2023-08-25T15:59:20.222538Z","iopub.status.idle":"2023-08-25T15:59:20.538289Z","shell.execute_reply.started":"2023-08-25T15:59:20.222500Z","shell.execute_reply":"2023-08-25T15:59:20.537148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df","metadata":{"executionInfo":{"elapsed":577,"status":"ok","timestamp":1692970042969,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"t6xxbg4yn9pT","outputId":"e296c9b8-69e1-4455-b2cd-e1b702d4f353","execution":{"iopub.status.busy":"2023-08-25T15:59:20.539897Z","iopub.execute_input":"2023-08-25T15:59:20.540358Z","iopub.status.idle":"2023-08-25T15:59:20.555002Z","shell.execute_reply.started":"2023-08-25T15:59:20.540314Z","shell.execute_reply":"2023-08-25T15:59:20.554020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df.prediction = test_preds","metadata":{"executionInfo":{"elapsed":1652,"status":"ok","timestamp":1692970076664,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"P1vWhwOIn4id","execution":{"iopub.status.busy":"2023-08-25T15:59:20.556222Z","iopub.execute_input":"2023-08-25T15:59:20.557152Z","iopub.status.idle":"2023-08-25T15:59:20.565030Z","shell.execute_reply.started":"2023-08-25T15:59:20.557116Z","shell.execute_reply":"2023-08-25T15:59:20.563719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df.prediction.value_counts()","metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1692970091815,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"ZSHxruEXoFWj","outputId":"ff871395-67c1-46e5-864b-076a46af45c0","execution":{"iopub.status.busy":"2023-08-25T15:59:20.567593Z","iopub.execute_input":"2023-08-25T15:59:20.568144Z","iopub.status.idle":"2023-08-25T15:59:20.587678Z","shell.execute_reply.started":"2023-08-25T15:59:20.568102Z","shell.execute_reply":"2023-08-25T15:59:20.586450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df.to_csv('submission.csv',index=None)","metadata":{"executionInfo":{"elapsed":1692,"status":"ok","timestamp":1692970132260,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"r_7JT4-JoGkL","execution":{"iopub.status.busy":"2023-08-25T15:59:20.588892Z","iopub.execute_input":"2023-08-25T15:59:20.589606Z","iopub.status.idle":"2023-08-25T15:59:21.235500Z","shell.execute_reply.started":"2023-08-25T15:59:20.589568Z","shell.execute_reply":"2023-08-25T15:59:21.234681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!head submission.csv","metadata":{"executionInfo":{"elapsed":16,"status":"ok","timestamp":1692970156968,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"496wLhWjoRA3","outputId":"f685c079-b604-40b7-870c-64cfb148b214","execution":{"iopub.status.busy":"2023-08-25T15:59:21.236799Z","iopub.execute_input":"2023-08-25T15:59:21.237736Z","iopub.status.idle":"2023-08-25T15:59:22.302031Z","shell.execute_reply.started":"2023-08-25T15:59:21.237693Z","shell.execute_reply":"2023-08-25T15:59:22.300425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = pd.read_csv('/kaggle/working/submission.csv')\nsub","metadata":{"executionInfo":{"elapsed":48,"status":"ok","timestamp":1692965499045,"user":{"displayName":"Kaushik kr. Sarma","userId":"04703072292199386040"},"user_tz":-330},"id":"o84S545HoWE4","execution":{"iopub.status.busy":"2023-08-25T16:33:25.653100Z","iopub.execute_input":"2023-08-25T16:33:25.653539Z","iopub.status.idle":"2023-08-25T16:33:26.013093Z","shell.execute_reply.started":"2023-08-25T16:33:25.653496Z","shell.execute_reply":"2023-08-25T16:33:26.011767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}