{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Text Classification with Bag of Words - Natural Language Processing\n\n![](https://i.imgur.com/hlEQ5X8.png)\n\n> _\"Natural language processing (NLP) is a subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data.\"_ - Wikipedia\n\n> _**Bag of Words**: The bag-of-words (BOW) model is a representation that turns arbitrary text into fixed-length vectors by counting how many times each word appears._\n \nOutline:\n\n1. Download and explore a real-world dataset\n2. Apply text preprocessing techniques\n3. Implement the bag of words model\n4. Train ML models for text classification\n5. Make predictions and submit to Kaggle\n\n\nDataset: https://www.kaggle.com/c/quora-insincere-questions-classification\n\n","metadata":{"id":"tBxC0flTQDd3"}},{"cell_type":"markdown","source":"## Download and Explore the Data\n\nOutline:\n\n1. Download the dataset from Kaggle to Colab\n2. Explore the data using Pandas\n3. Create a small working sample","metadata":{"id":"ZOcWwlvxQoSJ"}},{"cell_type":"markdown","source":"### Download the Data\n\n","metadata":{"id":"xCBkYxdzRPPq"}},{"cell_type":"code","source":"data = '../input/quora-insincere-questions-classification'\ntrain = data + '/train.csv'\ntest = data + '/test.csv'\nsubmission = data + '/sample_submission.csv'","metadata":{"executionInfo":{"elapsed":1627,"status":"ok","timestamp":1667911859298,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"Y-gnbjWgRg13","outputId":"c2277c8e-c5b9-4ebf-ac12-76939826bc07","execution":{"iopub.status.busy":"2022-11-17T11:50:12.145441Z","iopub.execute_input":"2022-11-17T11:50:12.145938Z","iopub.status.idle":"2022-11-17T11:50:12.157111Z","shell.execute_reply.started":"2022-11-17T11:50:12.145905Z","shell.execute_reply":"2022-11-17T11:50:12.155690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Explore the Data using Pandas","metadata":{"id":"Ql6Gv4shTSBJ"}},{"cell_type":"code","source":"import pandas as pd\nraw_df = pd.read_csv(train)\nraw_df","metadata":{"executionInfo":{"elapsed":358,"status":"ok","timestamp":1667912043319,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"pf65NRzdQ-hU","execution":{"iopub.status.busy":"2022-11-17T11:50:15.494318Z","iopub.execute_input":"2022-11-17T11:50:15.494747Z","iopub.status.idle":"2022-11-17T11:50:18.555279Z","shell.execute_reply.started":"2022-11-17T11:50:15.494714Z","shell.execute_reply":"2022-11-17T11:50:18.554019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sincere_df = raw_df[raw_df.target == 0]\n","metadata":{"executionInfo":{"elapsed":4754,"status":"ok","timestamp":1667912065510,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"H1FjN5gDTfUY","execution":{"iopub.status.busy":"2022-11-17T11:50:25.203868Z","iopub.execute_input":"2022-11-17T11:50:25.204260Z","iopub.status.idle":"2022-11-17T11:50:25.309174Z","shell.execute_reply.started":"2022-11-17T11:50:25.204228Z","shell.execute_reply":"2022-11-17T11:50:25.307690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sincere_df.question_text.values[:10]\n","metadata":{"executionInfo":{"elapsed":474,"status":"ok","timestamp":1667912069885,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"CHMgt_5iTfRt","outputId":"7ad71a7a-b478-4a42-bce0-ee467479480f","execution":{"iopub.status.busy":"2022-11-17T11:50:26.303973Z","iopub.execute_input":"2022-11-17T11:50:26.304372Z","iopub.status.idle":"2022-11-17T11:50:26.311877Z","shell.execute_reply.started":"2022-11-17T11:50:26.304342Z","shell.execute_reply":"2022-11-17T11:50:26.310622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"insincere_df = raw_df[raw_df.target == 1]\n","metadata":{"executionInfo":{"elapsed":382,"status":"ok","timestamp":1667912255712,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"wV2xGZ3STfPx","execution":{"iopub.status.busy":"2022-11-17T11:50:27.424719Z","iopub.execute_input":"2022-11-17T11:50:27.425259Z","iopub.status.idle":"2022-11-17T11:50:27.458708Z","shell.execute_reply.started":"2022-11-17T11:50:27.425221Z","shell.execute_reply":"2022-11-17T11:50:27.457074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"insincere_df.question_text.values[:10]\n","metadata":{"executionInfo":{"elapsed":375,"status":"ok","timestamp":1667912261902,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"me4srOFdUMCQ","outputId":"6f5e9340-bdb2-4c22-b75e-57177ccf6ea9","execution":{"iopub.status.busy":"2022-11-17T11:50:29.184432Z","iopub.execute_input":"2022-11-17T11:50:29.184824Z","iopub.status.idle":"2022-11-17T11:50:29.194933Z","shell.execute_reply.started":"2022-11-17T11:50:29.184793Z","shell.execute_reply":"2022-11-17T11:50:29.193076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"raw_df.target.value_counts()\n","metadata":{"executionInfo":{"elapsed":572,"status":"ok","timestamp":1667912293701,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"cTApBtxIUMAJ","execution":{"iopub.status.busy":"2022-11-17T11:50:30.344252Z","iopub.execute_input":"2022-11-17T11:50:30.345711Z","iopub.status.idle":"2022-11-17T11:50:30.368390Z","shell.execute_reply.started":"2022-11-17T11:50:30.345652Z","shell.execute_reply":"2022-11-17T11:50:30.366354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"raw_df.target.value_counts(normalize=True)\n","metadata":{"executionInfo":{"elapsed":652,"status":"ok","timestamp":1667912301181,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"-nFN5bZVUL91","outputId":"ce551e6c-3255-408d-9182-4408269d67bf","execution":{"iopub.status.busy":"2022-11-17T11:50:31.825466Z","iopub.execute_input":"2022-11-17T11:50:31.825924Z","iopub.status.idle":"2022-11-17T11:50:31.862076Z","shell.execute_reply.started":"2022-11-17T11:50:31.825890Z","shell.execute_reply":"2022-11-17T11:50:31.860585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"raw_df.target.value_counts(normalize=True).plot(kind='bar')\n","metadata":{"executionInfo":{"elapsed":431,"status":"ok","timestamp":1667912372012,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"n2VQSfKmUn7Z","outputId":"abe27eed-82e0-472d-f0bb-3c28a5a77b06","execution":{"iopub.status.busy":"2022-11-17T11:50:34.044595Z","iopub.execute_input":"2022-11-17T11:50:34.045112Z","iopub.status.idle":"2022-11-17T11:50:34.259523Z","shell.execute_reply.started":"2022-11-17T11:50:34.045074Z","shell.execute_reply":"2022-11-17T11:50:34.258299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = pd.read_csv(test)\n","metadata":{"executionInfo":{"elapsed":395,"status":"ok","timestamp":1667912455099,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ZSzHylWuU0MF","outputId":"557af698-754b-49e6-a8f7-a504aaa74737","execution":{"iopub.status.busy":"2022-11-17T11:50:35.094611Z","iopub.execute_input":"2022-11-17T11:50:35.095116Z","iopub.status.idle":"2022-11-17T11:50:36.176741Z","shell.execute_reply.started":"2022-11-17T11:50:35.095079Z","shell.execute_reply":"2022-11-17T11:50:36.175445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df\n","metadata":{"executionInfo":{"elapsed":1646,"status":"ok","timestamp":1667912505443,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"_lA6Vb6YVN1p","execution":{"iopub.status.busy":"2022-11-17T11:50:36.194164Z","iopub.execute_input":"2022-11-17T11:50:36.194578Z","iopub.status.idle":"2022-11-17T11:50:36.209561Z","shell.execute_reply.started":"2022-11-17T11:50:36.194543Z","shell.execute_reply":"2022-11-17T11:50:36.208120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df = pd.read_csv(submission)\nsub_df","metadata":{"executionInfo":{"elapsed":3,"status":"ok","timestamp":1667912507665,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"gH0ij_pmVNzV","outputId":"ad30dad3-ffd4-43ac-c04d-659ac7cf006d","execution":{"iopub.status.busy":"2022-11-17T11:50:38.149559Z","iopub.execute_input":"2022-11-17T11:50:38.149954Z","iopub.status.idle":"2022-11-17T11:50:38.434907Z","shell.execute_reply.started":"2022-11-17T11:50:38.149924Z","shell.execute_reply":"2022-11-17T11:50:38.433526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df.prediction.value_counts()\n","metadata":{"executionInfo":{"elapsed":532,"status":"ok","timestamp":1667912554978,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"exDbcT9bVNw0","execution":{"iopub.status.busy":"2022-11-17T11:50:39.594654Z","iopub.execute_input":"2022-11-17T11:50:39.595034Z","iopub.status.idle":"2022-11-17T11:50:39.609245Z","shell.execute_reply.started":"2022-11-17T11:50:39.595005Z","shell.execute_reply":"2022-11-17T11:50:39.607336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create a Working Sample\n","metadata":{"id":"v7sWG0KCV0g-"}},{"cell_type":"code","source":"import os\nIS_KAGGLE = 'KAGGLE_KERNEL_RUN_TYPE' in os.environ","metadata":{"executionInfo":{"elapsed":653,"status":"ok","timestamp":1667912690959,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"hu5s7hLKV7Kr","execution":{"iopub.status.busy":"2022-11-17T11:50:42.434362Z","iopub.execute_input":"2022-11-17T11:50:42.434805Z","iopub.status.idle":"2022-11-17T11:50:42.440789Z","shell.execute_reply.started":"2022-11-17T11:50:42.434770Z","shell.execute_reply":"2022-11-17T11:50:42.439506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if IS_KAGGLE:\n    SAMPLE_SIZE = len(raw_df)\nelse:\n    SAMPLE_SIZE = 100_000\n","metadata":{"executionInfo":{"elapsed":702,"status":"ok","timestamp":1667912713848,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"0xzqyYo_V31J","execution":{"iopub.status.busy":"2022-11-17T11:50:44.234871Z","iopub.execute_input":"2022-11-17T11:50:44.235390Z","iopub.status.idle":"2022-11-17T11:50:44.241996Z","shell.execute_reply.started":"2022-11-17T11:50:44.235349Z","shell.execute_reply":"2022-11-17T11:50:44.240200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_df = raw_df.sample(SAMPLE_SIZE, random_state=42)\n","metadata":{"executionInfo":{"elapsed":886,"status":"ok","timestamp":1667912722831,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"4xFSOIfdV3yh","outputId":"a1f8d588-1005-475a-bfa9-04df72110179","execution":{"iopub.status.busy":"2022-11-17T11:50:45.664947Z","iopub.execute_input":"2022-11-17T11:50:45.666085Z","iopub.status.idle":"2022-11-17T11:50:46.131492Z","shell.execute_reply.started":"2022-11-17T11:50:45.666011Z","shell.execute_reply":"2022-11-17T11:50:46.129878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_df","metadata":{"execution":{"iopub.status.busy":"2022-11-17T11:50:47.464497Z","iopub.execute_input":"2022-11-17T11:50:47.464912Z","iopub.status.idle":"2022-11-17T11:50:47.478340Z","shell.execute_reply.started":"2022-11-17T11:50:47.464882Z","shell.execute_reply":"2022-11-17T11:50:47.477016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Text Preprocessing Techniques\n\nOutline:\n\n1. Understand the bag of words model\n2. Tokenization\n3. Stop word removal\n4. Stemming","metadata":{"id":"-gZy068pQoPg"}},{"cell_type":"markdown","source":"### Bag of Words Intuition\n\n1. Create a list of all the words across all the text documents\n2. You convert each document into vector counts of each word\n\n\nLimitations:\n1. There may be too many words in the dataset\n2. Some words may occur too frequently\n3. Some words may occur very rarely or only once\n4. A single word may have many forms (go, gone, going or bird vs. birds)","metadata":{"id":"pJ0mh3gjXH9e"}},{"cell_type":"code","source":"q0 = sincere_df.question_text.values[1]\nq0\n","metadata":{"executionInfo":{"elapsed":628,"status":"ok","timestamp":1667913361214,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"PDb6D4pfYWPu","execution":{"iopub.status.busy":"2022-11-17T11:50:51.324963Z","iopub.execute_input":"2022-11-17T11:50:51.325429Z","iopub.status.idle":"2022-11-17T11:50:51.335576Z","shell.execute_reply.started":"2022-11-17T11:50:51.325395Z","shell.execute_reply":"2022-11-17T11:50:51.333719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1 = raw_df[raw_df.target == 1].question_text.values[0]\nq1","metadata":{"executionInfo":{"elapsed":546,"status":"ok","timestamp":1667913367151,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"mQjFxTKFYjJk","outputId":"3012387d-9827-46bf-a3bb-d9d077ee6a9f","execution":{"iopub.status.busy":"2022-11-17T11:50:52.844900Z","iopub.execute_input":"2022-11-17T11:50:52.845840Z","iopub.status.idle":"2022-11-17T11:50:52.878212Z","shell.execute_reply.started":"2022-11-17T11:50:52.845795Z","shell.execute_reply":"2022-11-17T11:50:52.876458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Tokenization\n\nsplitting a document into words and separators","metadata":{"id":"qj4Zg1yaYPJv"}},{"cell_type":"code","source":"import nltk\nfrom nltk.tokenize import word_tokenize\nnltk.download('punkt')","metadata":{"executionInfo":{"elapsed":474,"status":"ok","timestamp":1667913549193,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"EqacxiuyZPds","execution":{"iopub.status.busy":"2022-11-17T11:50:55.074335Z","iopub.execute_input":"2022-11-17T11:50:55.074725Z","iopub.status.idle":"2022-11-17T11:50:55.083576Z","shell.execute_reply.started":"2022-11-17T11:50:55.074694Z","shell.execute_reply":"2022-11-17T11:50:55.082137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" q0\n","metadata":{"executionInfo":{"elapsed":1852,"status":"ok","timestamp":1667913505183,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"2vQDMEq0Q_lP","execution":{"iopub.status.busy":"2022-11-17T11:50:56.094593Z","iopub.execute_input":"2022-11-17T11:50:56.095034Z","iopub.status.idle":"2022-11-17T11:50:56.103040Z","shell.execute_reply.started":"2022-11-17T11:50:56.095002Z","shell.execute_reply":"2022-11-17T11:50:56.101494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"word_tokenize(q0)","metadata":{"executionInfo":{"elapsed":639,"status":"ok","timestamp":1667913551680,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"5eaAldKAZN-4","outputId":"d419ecd2-0540-4ed5-f0fc-32e68bd287dc","execution":{"iopub.status.busy":"2022-11-17T11:50:57.704830Z","iopub.execute_input":"2022-11-17T11:50:57.705252Z","iopub.status.idle":"2022-11-17T11:50:57.714396Z","shell.execute_reply.started":"2022-11-17T11:50:57.705221Z","shell.execute_reply":"2022-11-17T11:50:57.712795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1\n","metadata":{"executionInfo":{"elapsed":1019,"status":"ok","timestamp":1667913515768,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"VyIaEATyQ_ip","outputId":"655a30fa-ccf0-4839-aa2a-15f4437a5562","execution":{"iopub.status.busy":"2022-11-17T11:50:58.714638Z","iopub.execute_input":"2022-11-17T11:50:58.716030Z","iopub.status.idle":"2022-11-17T11:50:58.723999Z","shell.execute_reply.started":"2022-11-17T11:50:58.715977Z","shell.execute_reply":"2022-11-17T11:50:58.722599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"word_tokenize(q1)","metadata":{"executionInfo":{"elapsed":458,"status":"ok","timestamp":1667913557740,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"1yULG5nVQ_gk","outputId":"83e3941f-3d29-4008-c885-b9a5eaf93912","execution":{"iopub.status.busy":"2022-11-17T11:50:59.614623Z","iopub.execute_input":"2022-11-17T11:50:59.615101Z","iopub.status.idle":"2022-11-17T11:50:59.623307Z","shell.execute_reply.started":"2022-11-17T11:50:59.615063Z","shell.execute_reply":"2022-11-17T11:50:59.622222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_tok = word_tokenize(q0)\nq1_tok = word_tokenize(q1)","metadata":{"executionInfo":{"elapsed":451,"status":"ok","timestamp":1667913596244,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"gJ_lrgEpZIt8","outputId":"aa250716-93b6-47ab-cbd1-d9ba9076c620","execution":{"iopub.status.busy":"2022-11-17T11:51:00.544861Z","iopub.execute_input":"2022-11-17T11:51:00.545549Z","iopub.status.idle":"2022-11-17T11:51:00.552077Z","shell.execute_reply.started":"2022-11-17T11:51:00.545490Z","shell.execute_reply":"2022-11-17T11:51:00.550489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Stop Word Removal\n\nRemoving commonly occuring words","metadata":{"id":"Mf5aqvkGZrR-"}},{"cell_type":"code","source":"q1_tok\n","metadata":{"executionInfo":{"elapsed":385,"status":"ok","timestamp":1667913691780,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"7yZhlXV9Zxo-","outputId":"408a649d-3883-4f70-e1b5-a052ada07d60","execution":{"iopub.status.busy":"2022-11-17T11:51:02.224319Z","iopub.execute_input":"2022-11-17T11:51:02.224999Z","iopub.status.idle":"2022-11-17T11:51:02.233389Z","shell.execute_reply.started":"2022-11-17T11:51:02.224965Z","shell.execute_reply":"2022-11-17T11:51:02.231930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from nltk.corpus import stopwords\nnltk.download('stopwords')","metadata":{"executionInfo":{"elapsed":1009,"status":"ok","timestamp":1667913739715,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"s1qsENBhZrD0","execution":{"iopub.status.busy":"2022-11-17T11:51:03.089630Z","iopub.execute_input":"2022-11-17T11:51:03.090512Z","iopub.status.idle":"2022-11-17T11:51:03.100751Z","shell.execute_reply.started":"2022-11-17T11:51:03.090473Z","shell.execute_reply":"2022-11-17T11:51:03.099267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# comma will be added to every stopword that is why  - \", \".join\nenglish_stopwords = stopwords.words('english')\n\", \".join(english_stopwords)","metadata":{"executionInfo":{"elapsed":459,"status":"ok","timestamp":1667913791292,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"4kjpGIjraK7r","outputId":"dc16bc42-9e3e-4796-8937-5d9af34d099e","execution":{"iopub.status.busy":"2022-11-17T11:51:03.949634Z","iopub.execute_input":"2022-11-17T11:51:03.950352Z","iopub.status.idle":"2022-11-17T11:51:03.958224Z","shell.execute_reply.started":"2022-11-17T11:51:03.950315Z","shell.execute_reply":"2022-11-17T11:51:03.957076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def remove_stopwords(tokens):\n    return [word for word in tokens if word.lower() not in english_stopwords]\n","metadata":{"executionInfo":{"elapsed":430,"status":"ok","timestamp":1667913795278,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ejhkI9oVZrCD","execution":{"iopub.status.busy":"2022-11-17T11:51:04.674379Z","iopub.execute_input":"2022-11-17T11:51:04.675561Z","iopub.status.idle":"2022-11-17T11:51:04.681924Z","shell.execute_reply.started":"2022-11-17T11:51:04.675515Z","shell.execute_reply":"2022-11-17T11:51:04.680488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_tok","metadata":{"executionInfo":{"elapsed":357,"status":"ok","timestamp":1667913817359,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"jB5avxBsZqxB","outputId":"27c5b385-e8e1-4987-cc2a-19d43eb70308","execution":{"iopub.status.busy":"2022-11-17T11:51:05.254343Z","iopub.execute_input":"2022-11-17T11:51:05.254829Z","iopub.status.idle":"2022-11-17T11:51:05.264031Z","shell.execute_reply.started":"2022-11-17T11:51:05.254785Z","shell.execute_reply":"2022-11-17T11:51:05.262694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_stp = remove_stopwords(q0_tok)\nq0_stp","metadata":{"executionInfo":{"elapsed":352,"status":"ok","timestamp":1667913917513,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"7umWfNCvZqsF","execution":{"iopub.status.busy":"2022-11-17T11:51:06.644364Z","iopub.execute_input":"2022-11-17T11:51:06.644895Z","iopub.status.idle":"2022-11-17T11:51:06.652989Z","shell.execute_reply.started":"2022-11-17T11:51:06.644853Z","shell.execute_reply":"2022-11-17T11:51:06.651909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_stp = remove_stopwords(q1_tok)\n","metadata":{"executionInfo":{"elapsed":930,"status":"ok","timestamp":1667913931371,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"15NSoE_Saq6W","outputId":"4abcb5d0-fae7-4c59-f01d-c8d641d798e9","execution":{"iopub.status.busy":"2022-11-17T11:51:07.574603Z","iopub.execute_input":"2022-11-17T11:51:07.575880Z","iopub.status.idle":"2022-11-17T11:51:07.581882Z","shell.execute_reply.started":"2022-11-17T11:51:07.575826Z","shell.execute_reply":"2022-11-17T11:51:07.580275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_tok","metadata":{"executionInfo":{"elapsed":667,"status":"ok","timestamp":1667913944997,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"b59BPAClaq4B","execution":{"iopub.status.busy":"2022-11-17T11:51:08.324549Z","iopub.execute_input":"2022-11-17T11:51:08.325004Z","iopub.status.idle":"2022-11-17T11:51:08.334296Z","shell.execute_reply.started":"2022-11-17T11:51:08.324970Z","shell.execute_reply":"2022-11-17T11:51:08.332578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_stp\n","metadata":{"executionInfo":{"elapsed":502,"status":"ok","timestamp":1667913950071,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"UJpOKfd-Zqp_","outputId":"d9cd0cfc-26a5-45a2-eb18-3517d13bedda","execution":{"iopub.status.busy":"2022-11-17T11:51:09.255205Z","iopub.execute_input":"2022-11-17T11:51:09.256683Z","iopub.status.idle":"2022-11-17T11:51:09.266632Z","shell.execute_reply.started":"2022-11-17T11:51:09.256615Z","shell.execute_reply":"2022-11-17T11:51:09.265029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Stemming\n\n\"go\", \"gone\", \"going\" -> \"go\"\n\"birds\", \"bird\" -> \"bird\"","metadata":{"id":"FeVx68x5bHEF"}},{"cell_type":"code","source":"from nltk.stem.snowball import SnowballStemmer\nstemmer = SnowballStemmer(language='english')\n","metadata":{"executionInfo":{"elapsed":1054,"status":"ok","timestamp":1667914103321,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"9jFOfSqtbG6y","execution":{"iopub.status.busy":"2022-11-17T11:51:10.774626Z","iopub.execute_input":"2022-11-17T11:51:10.775370Z","iopub.status.idle":"2022-11-17T11:51:10.781437Z","shell.execute_reply.started":"2022-11-17T11:51:10.775319Z","shell.execute_reply":"2022-11-17T11:51:10.780030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stemmer.stem('going')","metadata":{"executionInfo":{"elapsed":544,"status":"ok","timestamp":1667914116697,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"SC0PMIy3bG4r","execution":{"iopub.status.busy":"2022-11-17T11:51:11.519377Z","iopub.execute_input":"2022-11-17T11:51:11.519778Z","iopub.status.idle":"2022-11-17T11:51:11.526734Z","shell.execute_reply.started":"2022-11-17T11:51:11.519744Z","shell.execute_reply":"2022-11-17T11:51:11.525469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_stm = [stemmer.stem(word) for word in q0_stp]\n","metadata":{"executionInfo":{"elapsed":792,"status":"ok","timestamp":1667914135778,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"apl0nhFqbG2F","outputId":"fc70d0f5-0f7a-49b4-ace3-ea1113cb8c66","execution":{"iopub.status.busy":"2022-11-17T11:51:12.305461Z","iopub.execute_input":"2022-11-17T11:51:12.306096Z","iopub.status.idle":"2022-11-17T11:51:12.313527Z","shell.execute_reply.started":"2022-11-17T11:51:12.306046Z","shell.execute_reply":"2022-11-17T11:51:12.311727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_stp\n","metadata":{"executionInfo":{"elapsed":386,"status":"ok","timestamp":1667914142907,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"wtIoHT1ybGz2","outputId":"3609cf3d-2c9f-4638-db1f-07ce31e63f20","execution":{"iopub.status.busy":"2022-11-17T11:51:13.095028Z","iopub.execute_input":"2022-11-17T11:51:13.095559Z","iopub.status.idle":"2022-11-17T11:51:13.103535Z","shell.execute_reply.started":"2022-11-17T11:51:13.095516Z","shell.execute_reply":"2022-11-17T11:51:13.102213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_stm","metadata":{"executionInfo":{"elapsed":364,"status":"ok","timestamp":1667914180406,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"hGdECu-rbmbZ","execution":{"iopub.status.busy":"2022-11-17T11:51:13.980062Z","iopub.execute_input":"2022-11-17T11:51:13.980593Z","iopub.status.idle":"2022-11-17T11:51:13.988288Z","shell.execute_reply.started":"2022-11-17T11:51:13.980553Z","shell.execute_reply":"2022-11-17T11:51:13.986985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_stm = [stemmer.stem(word) for word in q1_stp]\n","metadata":{"executionInfo":{"elapsed":817,"status":"ok","timestamp":1667914186316,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"BY0fs95pbqtH","outputId":"5e0d4927-385f-40c7-ad88-82ec30049890","execution":{"iopub.status.busy":"2022-11-17T11:51:14.924631Z","iopub.execute_input":"2022-11-17T11:51:14.926050Z","iopub.status.idle":"2022-11-17T11:51:14.932217Z","shell.execute_reply.started":"2022-11-17T11:51:14.925997Z","shell.execute_reply":"2022-11-17T11:51:14.930787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_stp\n","metadata":{"executionInfo":{"elapsed":573,"status":"ok","timestamp":1667914193742,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ARuFpGkHbqn2","outputId":"3e4e85e5-bb93-4feb-97f2-5141ebbcd1db","execution":{"iopub.status.busy":"2022-11-17T11:51:15.935225Z","iopub.execute_input":"2022-11-17T11:51:15.935735Z","iopub.status.idle":"2022-11-17T11:51:15.944373Z","shell.execute_reply.started":"2022-11-17T11:51:15.935697Z","shell.execute_reply":"2022-11-17T11:51:15.942536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_stm","metadata":{"executionInfo":{"elapsed":432,"status":"ok","timestamp":1667914233752,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"kiFiGdCpbrot","execution":{"iopub.status.busy":"2022-11-17T11:51:16.884207Z","iopub.execute_input":"2022-11-17T11:51:16.884629Z","iopub.status.idle":"2022-11-17T11:51:16.892845Z","shell.execute_reply.started":"2022-11-17T11:51:16.884594Z","shell.execute_reply":"2022-11-17T11:51:16.891608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Lemmatization\n\n\"love\" -> \"love\"\n\"loving\" -> \"love\"\n\"lovable\" -> \"love\"\n","metadata":{"id":"VWWkiTcWcAR7"}},{"cell_type":"markdown","source":"## Implement Bag of Words \n\n\nOutline:\n\n1. Create a vocabulary using Count Vectorizer\n2. Transform text to vectors using Count Vectorizer\n3. Configure text preprocessing in Count Vectorizer","metadata":{"id":"CuTR9sOWQoMq"}},{"cell_type":"markdown","source":"### Create a Vocabulary","metadata":{"id":"0HWShc6jdPFy"}},{"cell_type":"code","source":"sample_df\n","metadata":{"executionInfo":{"elapsed":419,"status":"ok","timestamp":1667914532328,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"uGScEAd9c_dQ","outputId":"e923813b-b8fd-438b-a6f0-743c280a204d","execution":{"iopub.status.busy":"2022-11-17T11:51:20.744094Z","iopub.execute_input":"2022-11-17T11:51:20.744775Z","iopub.status.idle":"2022-11-17T11:51:20.760720Z","shell.execute_reply.started":"2022-11-17T11:51:20.744728Z","shell.execute_reply":"2022-11-17T11:51:20.758995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_df = sample_df[:5]\nsmall_df","metadata":{"executionInfo":{"elapsed":707,"status":"ok","timestamp":1667914559612,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"IKj0BM2NRATm","execution":{"iopub.status.busy":"2022-11-17T11:51:21.885091Z","iopub.execute_input":"2022-11-17T11:51:21.885579Z","iopub.status.idle":"2022-11-17T11:51:21.901362Z","shell.execute_reply.started":"2022-11-17T11:51:21.885545Z","shell.execute_reply":"2022-11-17T11:51:21.899705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_df.question_text.values\n","metadata":{"executionInfo":{"elapsed":1222,"status":"ok","timestamp":1667914576774,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"awqkIOZEdKFu","outputId":"9c110405-71e1-4c72-c02c-dfbc1d5be43c","execution":{"iopub.status.busy":"2022-11-17T11:51:22.874540Z","iopub.execute_input":"2022-11-17T11:51:22.875037Z","iopub.status.idle":"2022-11-17T11:51:22.885088Z","shell.execute_reply.started":"2022-11-17T11:51:22.875001Z","shell.execute_reply":"2022-11-17T11:51:22.883495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer\nsmall_vect = CountVectorizer()\nsmall_vect.fit(small_df.question_text)\n","metadata":{"executionInfo":{"elapsed":430,"status":"ok","timestamp":1667914566540,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"XZOIXmC3RARh","outputId":"330c3a06-1368-40de-ae9c-40f0061e61d6","execution":{"iopub.status.busy":"2022-11-17T11:51:23.809727Z","iopub.execute_input":"2022-11-17T11:51:23.810182Z","iopub.status.idle":"2022-11-17T11:51:23.820106Z","shell.execute_reply.started":"2022-11-17T11:51:23.810131Z","shell.execute_reply":"2022-11-17T11:51:23.818850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_vect.get_feature_names_out()","metadata":{"executionInfo":{"elapsed":362,"status":"ok","timestamp":1667914669258,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"Yr3XgzBWdNNx","execution":{"iopub.status.busy":"2022-11-17T11:51:25.564813Z","iopub.execute_input":"2022-11-17T11:51:25.565311Z","iopub.status.idle":"2022-11-17T11:51:25.574993Z","shell.execute_reply.started":"2022-11-17T11:51:25.565273Z","shell.execute_reply":"2022-11-17T11:51:25.573218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Transform documents into Vectors","metadata":{"id":"WXsUWz_hd9KE"}},{"cell_type":"code","source":"vectors = small_vect.transform(small_df.question_text)\nvectors\n\n","metadata":{"executionInfo":{"elapsed":533,"status":"ok","timestamp":1667914847949,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"nZVZ3jIddkKT","execution":{"iopub.status.busy":"2022-11-17T11:51:27.994281Z","iopub.execute_input":"2022-11-17T11:51:27.994708Z","iopub.status.idle":"2022-11-17T11:51:28.003548Z","shell.execute_reply.started":"2022-11-17T11:51:27.994672Z","shell.execute_reply":"2022-11-17T11:51:28.002350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectors.shape","metadata":{"executionInfo":{"elapsed":638,"status":"ok","timestamp":1667914853853,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"MIGbsQEQdkHt","outputId":"1ad2eff7-d795-4c0e-ab60-f88ce7b4ed8c","execution":{"iopub.status.busy":"2022-11-17T11:51:29.044932Z","iopub.execute_input":"2022-11-17T11:51:29.045992Z","iopub.status.idle":"2022-11-17T11:51:29.054644Z","shell.execute_reply.started":"2022-11-17T11:51:29.045930Z","shell.execute_reply":"2022-11-17T11:51:29.053634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_df.question_text.values[0]\n","metadata":{"executionInfo":{"elapsed":630,"status":"ok","timestamp":1667914900662,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"wriwC-1DeZPA","outputId":"9b7a3cba-e4e6-4023-851b-7cb6b6029f40","execution":{"iopub.status.busy":"2022-11-17T11:51:30.635158Z","iopub.execute_input":"2022-11-17T11:51:30.636509Z","iopub.status.idle":"2022-11-17T11:51:30.645430Z","shell.execute_reply.started":"2022-11-17T11:51:30.636439Z","shell.execute_reply":"2022-11-17T11:51:30.644396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectors[0].toarray()","metadata":{"executionInfo":{"elapsed":495,"status":"ok","timestamp":1667914953574,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"bXtFEWjtekqt","outputId":"a4c659e8-4c86-478e-c827-ddbacde28bd9","execution":{"iopub.status.busy":"2022-11-17T11:51:31.734870Z","iopub.execute_input":"2022-11-17T11:51:31.735603Z","iopub.status.idle":"2022-11-17T11:51:31.743973Z","shell.execute_reply.started":"2022-11-17T11:51:31.735566Z","shell.execute_reply":"2022-11-17T11:51:31.742440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectors.toarray()\n","metadata":{"executionInfo":{"elapsed":982,"status":"ok","timestamp":1667915020439,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"etQ-T7_1e0vF","outputId":"9446f31f-ae80-4fac-857f-42f9481bafed","execution":{"iopub.status.busy":"2022-11-17T11:51:34.805015Z","iopub.execute_input":"2022-11-17T11:51:34.805669Z","iopub.status.idle":"2022-11-17T11:51:34.814869Z","shell.execute_reply.started":"2022-11-17T11:51:34.805621Z","shell.execute_reply":"2022-11-17T11:51:34.813426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Configure Count Vectorizer Parameters","metadata":{"id":"vZPCOwobe485"}},{"cell_type":"code","source":"stemmer = SnowballStemmer(language='english')\n","metadata":{"executionInfo":{"elapsed":391,"status":"ok","timestamp":1667915246398,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"xUjZDsdqfq4u","execution":{"iopub.status.busy":"2022-11-17T11:51:40.124662Z","iopub.execute_input":"2022-11-17T11:51:40.125260Z","iopub.status.idle":"2022-11-17T11:51:40.131800Z","shell.execute_reply.started":"2022-11-17T11:51:40.125209Z","shell.execute_reply":"2022-11-17T11:51:40.130890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def tokenize(text):\n    return [stemmer.stem(word) for word in word_tokenize(text)]\n","metadata":{"executionInfo":{"elapsed":368,"status":"ok","timestamp":1667915256240,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"9hZzn1LBfPoM","execution":{"iopub.status.busy":"2022-11-17T11:51:40.924248Z","iopub.execute_input":"2022-11-17T11:51:40.924674Z","iopub.status.idle":"2022-11-17T11:51:40.930290Z","shell.execute_reply.started":"2022-11-17T11:51:40.924610Z","shell.execute_reply":"2022-11-17T11:51:40.929126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tokenize('What is the really (dealing) here?')","metadata":{"executionInfo":{"elapsed":358,"status":"ok","timestamp":1667915263406,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"OAausoIrfa2I","outputId":"96c1e3a3-1f5c-40d0-e05b-1c32046aff8c","execution":{"iopub.status.busy":"2022-11-17T11:51:41.693852Z","iopub.execute_input":"2022-11-17T11:51:41.694296Z","iopub.status.idle":"2022-11-17T11:51:41.703560Z","shell.execute_reply.started":"2022-11-17T11:51:41.694262Z","shell.execute_reply":"2022-11-17T11:51:41.702077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectorizer = CountVectorizer(lowercase=True, \n                             tokenizer=tokenize,\n                             stop_words=english_stopwords,\n                             max_features=1000)\n","metadata":{"executionInfo":{"elapsed":559,"status":"ok","timestamp":1667915360958,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"pJGAErs7e9Fh","execution":{"iopub.status.busy":"2022-11-17T11:51:42.544268Z","iopub.execute_input":"2022-11-17T11:51:42.544983Z","iopub.status.idle":"2022-11-17T11:51:42.550426Z","shell.execute_reply.started":"2022-11-17T11:51:42.544949Z","shell.execute_reply":"2022-11-17T11:51:42.548938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%time\nvectorizer.fit(sample_df.question_text)","metadata":{"execution":{"iopub.status.busy":"2022-11-17T11:51:44.769060Z","iopub.execute_input":"2022-11-17T11:51:44.769488Z","iopub.status.idle":"2022-11-17T11:59:13.993576Z","shell.execute_reply.started":"2022-11-17T11:51:44.769456Z","shell.execute_reply":"2022-11-17T11:59:13.992330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(vectorizer.vocabulary_)\n","metadata":{"executionInfo":{"elapsed":2068,"status":"ok","timestamp":1667915449808,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"sJBlLm5Ye88Z","outputId":"e8ad80f8-cd4d-4289-fd26-163dc808e69f","execution":{"iopub.status.busy":"2022-11-17T12:10:30.848520Z","iopub.execute_input":"2022-11-17T12:10:30.848919Z","iopub.status.idle":"2022-11-17T12:10:30.858217Z","shell.execute_reply.started":"2022-11-17T12:10:30.848888Z","shell.execute_reply":"2022-11-17T12:10:30.856537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectorizer.get_feature_names_out()[:100]\n","metadata":{"executionInfo":{"elapsed":1059,"status":"ok","timestamp":1667915475499,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"b9K638h1gg5_","outputId":"955fe2e0-52ab-4137-9ecc-f18a64542e45","execution":{"iopub.status.busy":"2022-11-17T12:10:42.594669Z","iopub.execute_input":"2022-11-17T12:10:42.595308Z","iopub.status.idle":"2022-11-17T12:10:42.605882Z","shell.execute_reply.started":"2022-11-17T12:10:42.595258Z","shell.execute_reply":"2022-11-17T12:10:42.604077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ninputs = vectorizer.transform(sample_df.question_text)","metadata":{"executionInfo":{"elapsed":30784,"status":"ok","timestamp":1667915566857,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"n3iq65OsguTR","outputId":"35c22bcd-b6c2-4629-9e83-ad81a29456ee","execution":{"iopub.status.busy":"2022-11-17T12:11:09.838645Z","iopub.execute_input":"2022-11-17T12:11:09.839062Z","iopub.status.idle":"2022-11-17T12:18:51.501263Z","shell.execute_reply.started":"2022-11-17T12:11:09.839030Z","shell.execute_reply":"2022-11-17T12:18:51.499979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"inputs.shape\n","metadata":{"executionInfo":{"elapsed":661,"status":"ok","timestamp":1667915574286,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"9aEbKIC0gg3o","outputId":"a0f9fdcf-a53d-455d-bd49-691f1f737a47","execution":{"iopub.status.busy":"2022-11-17T12:20:38.308294Z","iopub.execute_input":"2022-11-17T12:20:38.308690Z","iopub.status.idle":"2022-11-17T12:20:38.315072Z","shell.execute_reply.started":"2022-11-17T12:20:38.308660Z","shell.execute_reply":"2022-11-17T12:20:38.314350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"inputs.shape\n","metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1667915578819,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"meslQK7Le86I","outputId":"d6cdab93-2640-43d0-d5fd-b907c201177d","execution":{"iopub.status.busy":"2022-11-17T12:20:48.518705Z","iopub.execute_input":"2022-11-17T12:20:48.519082Z","iopub.status.idle":"2022-11-17T12:20:48.526876Z","shell.execute_reply.started":"2022-11-17T12:20:48.519053Z","shell.execute_reply":"2022-11-17T12:20:48.525765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_df.question_text.values[0]\n","metadata":{"executionInfo":{"elapsed":652,"status":"ok","timestamp":1667915621850,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ARTJ68oehHVR","outputId":"83f27c51-7c51-4808-9450-489e5599776b","execution":{"iopub.status.busy":"2022-11-17T12:20:59.507758Z","iopub.execute_input":"2022-11-17T12:20:59.508171Z","iopub.status.idle":"2022-11-17T12:20:59.517334Z","shell.execute_reply.started":"2022-11-17T12:20:59.508109Z","shell.execute_reply":"2022-11-17T12:20:59.515921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df\n","metadata":{"executionInfo":{"elapsed":1081,"status":"ok","timestamp":1667915683093,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"fJ497jJyhCaK","outputId":"cdb77624-843c-4fe1-ff52-5f93794be092","execution":{"iopub.status.busy":"2022-11-17T12:21:09.637750Z","iopub.execute_input":"2022-11-17T12:21:09.638159Z","iopub.status.idle":"2022-11-17T12:21:09.654748Z","shell.execute_reply.started":"2022-11-17T12:21:09.638112Z","shell.execute_reply":"2022-11-17T12:21:09.653229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntest_inputs = vectorizer.transform(test_df.question_text)","metadata":{"executionInfo":{"elapsed":112943,"status":"ok","timestamp":1667915815920,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"UDu0wUf9g--d","outputId":"3db731b6-8002-435a-bfb5-7500c8921da8","execution":{"iopub.status.busy":"2022-11-17T12:21:23.257542Z","iopub.execute_input":"2022-11-17T12:21:23.258001Z","iopub.status.idle":"2022-11-17T12:23:34.173438Z","shell.execute_reply.started":"2022-11-17T12:21:23.257968Z","shell.execute_reply":"2022-11-17T12:23:34.172195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ML Models for Text Classification\n\nOutline:\n\n- Create a training & validation set\n- Train a logistic regression model\n- Make predictions on training, validation & test data","metadata":{"id":"AyVwmOboQoJn"}},{"cell_type":"markdown","source":"### Split into Training and Validation Set","metadata":{"id":"1CPsktK-iPgK"}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n","metadata":{"executionInfo":{"elapsed":3,"status":"ok","timestamp":1667915931527,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"hoIv721biUVy","outputId":"73a07b8c-ec27-495e-8ccd-9bb4c53e40da","execution":{"iopub.status.busy":"2022-11-17T12:25:49.389053Z","iopub.execute_input":"2022-11-17T12:25:49.389586Z","iopub.status.idle":"2022-11-17T12:25:49.395717Z","shell.execute_reply.started":"2022-11-17T12:25:49.389543Z","shell.execute_reply":"2022-11-17T12:25:49.394613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_inputs, val_inputs, train_targets, val_targets = train_test_split(inputs, sample_df.target,\n                                                                        test_size=0.3, random_state=42)\n\n","metadata":{"execution":{"iopub.status.busy":"2022-11-17T12:27:11.918342Z","iopub.execute_input":"2022-11-17T12:27:11.919708Z","iopub.status.idle":"2022-11-17T12:27:12.271230Z","shell.execute_reply.started":"2022-11-17T12:27:11.919619Z","shell.execute_reply":"2022-11-17T12:27:12.270059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_inputs.shape\n","metadata":{"executionInfo":{"elapsed":613,"status":"ok","timestamp":1667915949798,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ckPyKMcMRBET","outputId":"0bb74251-5275-42ed-b60a-eb763113e3c0","execution":{"iopub.status.busy":"2022-11-17T12:27:34.208713Z","iopub.execute_input":"2022-11-17T12:27:34.209281Z","iopub.status.idle":"2022-11-17T12:27:34.217510Z","shell.execute_reply.started":"2022-11-17T12:27:34.209238Z","shell.execute_reply":"2022-11-17T12:27:34.216361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_targets.shape","metadata":{"executionInfo":{"elapsed":348,"status":"ok","timestamp":1667916033391,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"OB2jqu7dirOn","execution":{"iopub.status.busy":"2022-11-17T12:27:36.827526Z","iopub.execute_input":"2022-11-17T12:27:36.828137Z","iopub.status.idle":"2022-11-17T12:27:36.838934Z","shell.execute_reply.started":"2022-11-17T12:27:36.828086Z","shell.execute_reply":"2022-11-17T12:27:36.837010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_inputs.shape\n","metadata":{"executionInfo":{"elapsed":364,"status":"ok","timestamp":1667916089349,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"m3eE_6y6ieLP","execution":{"iopub.status.busy":"2022-11-17T12:28:05.352404Z","iopub.execute_input":"2022-11-17T12:28:05.352808Z","iopub.status.idle":"2022-11-17T12:28:05.360652Z","shell.execute_reply.started":"2022-11-17T12:28:05.352776Z","shell.execute_reply":"2022-11-17T12:28:05.359319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_targets.shape","metadata":{"executionInfo":{"elapsed":908,"status":"ok","timestamp":1667916115211,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"1hMoWlcrRBBt","outputId":"c604cdcc-ed79-4c04-dba0-8d46713cab38","execution":{"iopub.status.busy":"2022-11-17T12:28:08.507161Z","iopub.execute_input":"2022-11-17T12:28:08.507607Z","iopub.status.idle":"2022-11-17T12:28:08.514918Z","shell.execute_reply.started":"2022-11-17T12:28:08.507568Z","shell.execute_reply":"2022-11-17T12:28:08.513760Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train Logistic Regression model","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nMAX_ITER = 1000\nmodel = LogisticRegression(max_iter=MAX_ITER, solver='sag')\n","metadata":{"executionInfo":{"elapsed":986,"status":"ok","timestamp":1667916129267,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ioOd41xKjCkX","outputId":"f3c280ca-aa5c-4f49-d551-3ffebe05c89a","execution":{"iopub.status.busy":"2022-11-17T12:28:33.347624Z","iopub.execute_input":"2022-11-17T12:28:33.348084Z","iopub.status.idle":"2022-11-17T12:28:33.354604Z","shell.execute_reply.started":"2022-11-17T12:28:33.348048Z","shell.execute_reply":"2022-11-17T12:28:33.352973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nmodel.fit(train_inputs, train_targets)","metadata":{"executionInfo":{"elapsed":493,"status":"ok","timestamp":1667916138104,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"d2ipJkxRjChw","outputId":"798265ae-8305-41b5-f96c-8611c45aef4f","execution":{"iopub.status.busy":"2022-11-17T12:28:42.569695Z","iopub.execute_input":"2022-11-17T12:28:42.570716Z","iopub.status.idle":"2022-11-17T12:38:08.054269Z","shell.execute_reply.started":"2022-11-17T12:28:42.570662Z","shell.execute_reply":"2022-11-17T12:38:08.052874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Make predictions using the model","metadata":{"id":"o16-8HmXmQT0"}},{"cell_type":"code","source":"train_preds = model.predict(train_inputs)\n","metadata":{"executionInfo":{"elapsed":1020,"status":"ok","timestamp":1667916460185,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"i12DBlkVkThU","execution":{"iopub.status.busy":"2022-11-17T12:40:18.368080Z","iopub.execute_input":"2022-11-17T12:40:18.368518Z","iopub.status.idle":"2022-11-17T12:40:18.400642Z","shell.execute_reply.started":"2022-11-17T12:40:18.368475Z","shell.execute_reply":"2022-11-17T12:40:18.399389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_targets","metadata":{"executionInfo":{"elapsed":363,"status":"ok","timestamp":1667916573761,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"urNCyppPkxZB","outputId":"71349a02-b3a6-4b44-d87f-727dde95605e","execution":{"iopub.status.busy":"2022-11-17T12:40:21.667331Z","iopub.execute_input":"2022-11-17T12:40:21.667724Z","iopub.status.idle":"2022-11-17T12:40:21.678795Z","shell.execute_reply.started":"2022-11-17T12:40:21.667694Z","shell.execute_reply":"2022-11-17T12:40:21.677267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_preds\n","metadata":{"executionInfo":{"elapsed":861,"status":"ok","timestamp":1667916464475,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"qS8PDYxljNDP","outputId":"9216c5a9-c9aa-4d6a-d540-e804d4d3c925","execution":{"iopub.status.busy":"2022-11-17T12:40:35.576925Z","iopub.execute_input":"2022-11-17T12:40:35.577363Z","iopub.status.idle":"2022-11-17T12:40:35.586272Z","shell.execute_reply.started":"2022-11-17T12:40:35.577329Z","shell.execute_reply":"2022-11-17T12:40:35.585071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series(train_preds).value_counts()\n","metadata":{"executionInfo":{"elapsed":567,"status":"ok","timestamp":1667916484628,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"W74eTiJgjNA9","outputId":"4f1aa29b-d959-4561-9384-a83fc36ca13d","execution":{"iopub.status.busy":"2022-11-17T12:40:43.828094Z","iopub.execute_input":"2022-11-17T12:40:43.828477Z","iopub.status.idle":"2022-11-17T12:40:43.846780Z","shell.execute_reply.started":"2022-11-17T12:40:43.828447Z","shell.execute_reply":"2022-11-17T12:40:43.845214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series(train_targets).value_counts()\n","metadata":{"executionInfo":{"elapsed":1125,"status":"ok","timestamp":1667916512008,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"pixsvLxrkhK4","outputId":"44819bce-587f-41e3-cb6b-55e67b8c7c84","execution":{"iopub.status.busy":"2022-11-17T12:40:53.347208Z","iopub.execute_input":"2022-11-17T12:40:53.347600Z","iopub.status.idle":"2022-11-17T12:40:53.366458Z","shell.execute_reply.started":"2022-11-17T12:40:53.347570Z","shell.execute_reply":"2022-11-17T12:40:53.365086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import accuracy_score\naccuracy_score(train_targets, train_preds)","metadata":{"executionInfo":{"elapsed":370,"status":"ok","timestamp":1667916547769,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"YxH33IpFkpXJ","execution":{"iopub.status.busy":"2022-11-17T12:41:07.468573Z","iopub.execute_input":"2022-11-17T12:41:07.469031Z","iopub.status.idle":"2022-11-17T12:41:07.566068Z","shell.execute_reply.started":"2022-11-17T12:41:07.468996Z","shell.execute_reply":"2022-11-17T12:41:07.564914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\naccuracy_score(train_targets, np.zeros(len(train_targets)))","metadata":{"executionInfo":{"elapsed":893,"status":"ok","timestamp":1667916561938,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"oUgKam9RkYD6","outputId":"dfc12873-4db8-4993-ad67-bbf6456d8b18","execution":{"iopub.status.busy":"2022-11-17T12:41:21.887237Z","iopub.execute_input":"2022-11-17T12:41:21.887652Z","iopub.status.idle":"2022-11-17T12:41:22.004385Z","shell.execute_reply.started":"2022-11-17T12:41:21.887619Z","shell.execute_reply":"2022-11-17T12:41:22.002985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import f1_score\nf1_score(train_targets, train_preds)","metadata":{"executionInfo":{"elapsed":373,"status":"ok","timestamp":1667916643295,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ipzg1inClCcJ","execution":{"iopub.status.busy":"2022-11-17T12:41:37.458575Z","iopub.execute_input":"2022-11-17T12:41:37.458951Z","iopub.status.idle":"2022-11-17T12:41:37.859183Z","shell.execute_reply.started":"2022-11-17T12:41:37.458922Z","shell.execute_reply":"2022-11-17T12:41:37.858034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f1_score(train_targets, np.zeros(len(train_targets)))","metadata":{"executionInfo":{"elapsed":620,"status":"ok","timestamp":1667916647313,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"_P-IsOtTk5x_","outputId":"3aed16ea-c1ab-4c6c-db22-cc083ebe1278","execution":{"iopub.status.busy":"2022-11-17T12:41:53.162485Z","iopub.execute_input":"2022-11-17T12:41:53.163606Z","iopub.status.idle":"2022-11-17T12:41:53.647047Z","shell.execute_reply.started":"2022-11-17T12:41:53.163552Z","shell.execute_reply":"2022-11-17T12:41:53.645723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_preds = np.random.choice((0, 1), len(train_targets))\nf1_score(train_targets, random_preds)","metadata":{"executionInfo":{"elapsed":896,"status":"ok","timestamp":1667916682226,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"89TtZhqclKwb","execution":{"iopub.status.busy":"2022-11-17T12:42:09.329693Z","iopub.execute_input":"2022-11-17T12:42:09.330565Z","iopub.status.idle":"2022-11-17T12:42:09.810485Z","shell.execute_reply.started":"2022-11-17T12:42:09.330521Z","shell.execute_reply":"2022-11-17T12:42:09.809331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_preds = model.predict(val_inputs)\n","metadata":{"executionInfo":{"elapsed":476,"status":"ok","timestamp":1667916695952,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"pim_sjENlKn0","outputId":"a919cf7d-3c6b-42ce-c99e-e2da856291ab","execution":{"iopub.status.busy":"2022-11-17T12:42:34.323700Z","iopub.execute_input":"2022-11-17T12:42:34.324125Z","iopub.status.idle":"2022-11-17T12:42:34.341610Z","shell.execute_reply.started":"2022-11-17T12:42:34.324079Z","shell.execute_reply":"2022-11-17T12:42:34.340633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_score(val_targets, val_preds)\n","metadata":{"executionInfo":{"elapsed":375,"status":"ok","timestamp":1667916720434,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"RembBU-Fksbk","outputId":"9c815608-6f06-45d9-c352-d52ad659c965","execution":{"iopub.status.busy":"2022-11-17T12:42:36.767303Z","iopub.execute_input":"2022-11-17T12:42:36.767745Z","iopub.status.idle":"2022-11-17T12:42:36.817813Z","shell.execute_reply.started":"2022-11-17T12:42:36.767712Z","shell.execute_reply":"2022-11-17T12:42:36.816310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f1_score(val_targets, val_preds)\n","metadata":{"executionInfo":{"elapsed":530,"status":"ok","timestamp":1667916804410,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"43hI8WB8ld0e","outputId":"36a47c48-a715-47a9-9698-92c5eec0cd7f","execution":{"iopub.status.busy":"2022-11-17T12:42:51.597554Z","iopub.execute_input":"2022-11-17T12:42:51.598027Z","iopub.status.idle":"2022-11-17T12:42:51.795548Z","shell.execute_reply.started":"2022-11-17T12:42:51.597995Z","shell.execute_reply":"2022-11-17T12:42:51.794157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sincere_df.question_text.values[:10]","metadata":{"executionInfo":{"elapsed":394,"status":"ok","timestamp":1667916889103,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"FM0X5ImAl7gB","execution":{"iopub.status.busy":"2022-11-17T12:42:54.417803Z","iopub.execute_input":"2022-11-17T12:42:54.418321Z","iopub.status.idle":"2022-11-17T12:42:54.427883Z","shell.execute_reply.started":"2022-11-17T12:42:54.418280Z","shell.execute_reply":"2022-11-17T12:42:54.426457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sincere_df.target.values[:10]\n","metadata":{"executionInfo":{"elapsed":592,"status":"ok","timestamp":1667916920449,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"WiXz18lal7db","outputId":"1199b1c7-51b2-4c95-d83e-d945e1541bf9","execution":{"iopub.status.busy":"2022-11-17T12:43:15.473606Z","iopub.execute_input":"2022-11-17T12:43:15.474058Z","iopub.status.idle":"2022-11-17T12:43:15.482548Z","shell.execute_reply.started":"2022-11-17T12:43:15.474019Z","shell.execute_reply":"2022-11-17T12:43:15.481266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.predict(vectorizer.transform(sincere_df.question_text.values[:10]))","metadata":{"executionInfo":{"elapsed":902,"status":"ok","timestamp":1667916932147,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"Kj-oIhIHksY-","outputId":"de4648a5-5f9c-48fb-f86e-aad601d0a956","execution":{"iopub.status.busy":"2022-11-17T12:43:18.727606Z","iopub.execute_input":"2022-11-17T12:43:18.728110Z","iopub.status.idle":"2022-11-17T12:43:18.742660Z","shell.execute_reply.started":"2022-11-17T12:43:18.728070Z","shell.execute_reply":"2022-11-17T12:43:18.741052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"insincere_df.question_text.values[:10]\n","metadata":{"executionInfo":{"elapsed":653,"status":"ok","timestamp":1667917017459,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"qxCSeXZQmHpS","outputId":"7cb895cb-7605-45d2-b650-b61b02cecffe","execution":{"iopub.status.busy":"2022-11-17T12:43:27.208318Z","iopub.execute_input":"2022-11-17T12:43:27.208814Z","iopub.status.idle":"2022-11-17T12:43:27.217794Z","shell.execute_reply.started":"2022-11-17T12:43:27.208778Z","shell.execute_reply":"2022-11-17T12:43:27.216212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"insincere_df.target.values[:10]\n","metadata":{"executionInfo":{"elapsed":1051,"status":"ok","timestamp":1667917031436,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"fwlbPtLDmgDf","outputId":"a6360495-7f8f-489e-b955-56fbcffa88bd","execution":{"iopub.status.busy":"2022-11-17T12:44:08.683474Z","iopub.execute_input":"2022-11-17T12:44:08.684008Z","iopub.status.idle":"2022-11-17T12:44:08.692857Z","shell.execute_reply.started":"2022-11-17T12:44:08.683969Z","shell.execute_reply":"2022-11-17T12:44:08.691699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.predict(vectorizer.transform(insincere_df.question_text.values[:10]))","metadata":{"executionInfo":{"elapsed":375,"status":"ok","timestamp":1667917062825,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"qHE1gR0mmV0l","outputId":"c1e5f7f9-e5a3-4ea7-ff65-caa1c9280ce0","execution":{"iopub.status.busy":"2022-11-17T12:44:12.498572Z","iopub.execute_input":"2022-11-17T12:44:12.499082Z","iopub.status.idle":"2022-11-17T12:44:12.516205Z","shell.execute_reply.started":"2022-11-17T12:44:12.499044Z","shell.execute_reply":"2022-11-17T12:44:12.514616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Make Predictions and Submit to Kaggle","metadata":{"id":"TSFA3r48Q2ec"}},{"cell_type":"code","source":"test_inputs.shape\n","metadata":{"executionInfo":{"elapsed":1103,"status":"ok","timestamp":1667917318912,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"bIAZhmcWQDd4","outputId":"e92eb5e4-41ad-4590-da74-d9b14dc9697a","execution":{"iopub.status.busy":"2022-11-17T12:44:47.868780Z","iopub.execute_input":"2022-11-17T12:44:47.870041Z","iopub.status.idle":"2022-11-17T12:44:47.877670Z","shell.execute_reply.started":"2022-11-17T12:44:47.869986Z","shell.execute_reply":"2022-11-17T12:44:47.876225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_preds = model.predict(test_inputs)","metadata":{"executionInfo":{"elapsed":460,"status":"ok","timestamp":1667917332470,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"jzb8x69DnoHz","outputId":"23a5695e-d7b4-49ae-c715-2360925ad007","execution":{"iopub.status.busy":"2022-11-17T12:44:52.857458Z","iopub.execute_input":"2022-11-17T12:44:52.858175Z","iopub.status.idle":"2022-11-17T12:44:52.878024Z","shell.execute_reply.started":"2022-11-17T12:44:52.858100Z","shell.execute_reply":"2022-11-17T12:44:52.877076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df.prediction = test_preds\n","metadata":{"executionInfo":{"elapsed":1112,"status":"ok","timestamp":1667917351071,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"oiY8Ai-anqRP","execution":{"iopub.status.busy":"2022-11-17T12:45:13.648168Z","iopub.execute_input":"2022-11-17T12:45:13.648595Z","iopub.status.idle":"2022-11-17T12:45:13.656083Z","shell.execute_reply.started":"2022-11-17T12:45:13.648562Z","shell.execute_reply":"2022-11-17T12:45:13.654761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df.prediction.value_counts()","metadata":{"executionInfo":{"elapsed":558,"status":"ok","timestamp":1667917386054,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"W8h1xhbZnwBt","outputId":"6698c104-0bd3-4c8a-b12f-6e59feac67a3","execution":{"iopub.status.busy":"2022-11-17T12:45:14.693761Z","iopub.execute_input":"2022-11-17T12:45:14.694427Z","iopub.status.idle":"2022-11-17T12:45:14.708579Z","shell.execute_reply.started":"2022-11-17T12:45:14.694376Z","shell.execute_reply":"2022-11-17T12:45:14.707237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df\n","metadata":{"executionInfo":{"elapsed":4,"status":"ok","timestamp":1667917429135,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"t6xxbg4yn9pT","execution":{"iopub.status.busy":"2022-11-17T12:45:25.177560Z","iopub.execute_input":"2022-11-17T12:45:25.177967Z","iopub.status.idle":"2022-11-17T12:45:25.192028Z","shell.execute_reply.started":"2022-11-17T12:45:25.177933Z","shell.execute_reply":"2022-11-17T12:45:25.190710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df.to_csv('submission.csv', index=None)\n","metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1667917437994,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"P1vWhwOIn4id","outputId":"4822357f-2f37-4a73-c939-ca1d3f981454","execution":{"iopub.status.busy":"2022-11-17T13:21:33.532534Z","iopub.execute_input":"2022-11-17T13:21:33.532976Z","iopub.status.idle":"2022-11-17T13:21:34.057032Z","shell.execute_reply.started":"2022-11-17T13:21:33.532939Z","shell.execute_reply":"2022-11-17T13:21:34.055429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!head submission.csv\n","metadata":{"executionInfo":{"elapsed":371,"status":"ok","timestamp":1667917442478,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ZSHxruEXoFWj","outputId":"231bcd4a-0219-46cc-935a-a83b4e501672","execution":{"iopub.status.busy":"2022-11-17T13:26:22.151078Z","iopub.execute_input":"2022-11-17T13:26:22.151639Z","iopub.status.idle":"2022-11-17T13:26:23.290322Z","shell.execute_reply.started":"2022-11-17T13:26:22.151599Z","shell.execute_reply":"2022-11-17T13:26:23.288662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":771,"status":"ok","timestamp":1667917485705,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"r_7JT4-JoGkL"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"executionInfo":{"elapsed":1077,"status":"ok","timestamp":1667917506958,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"496wLhWjoRA3","outputId":"24c74944-ee72-4168-8342-0c52decb77c4"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"o84S545HoWE4"},"execution_count":null,"outputs":[]}]}