{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Text Classification with Bag of Words - Natural Language Processing\n\n![](https://i.imgur.com/hlEQ5X8.png)\n\n> _\"Natural language processing (NLP) is a subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data.\"_ - Wikipedia\n\n> _**Bag of Words**: The bag-of-words (BOW) model is a representation that turns arbitrary text into fixed-length vectors by counting how many times each word appears._\n \nOutline:\n\n1. Download and explore a real-world dataset\n2. Apply text preprocessing techniques\n3. Implement the bag of words model\n4. Train ML models for text classification\n5. Make predictions and submit to Kaggle\n\n\nDataset: https://www.kaggle.com/c/quora-insincere-questions-classification\n\n","metadata":{"id":"tBxC0flTQDd3"}},{"cell_type":"markdown","source":"## Download and Explore the Data\n\nOutline:\n\n1. Download the dataset from Kaggle to Colab\n2. Explore the data using Pandas\n3. Create a small working sample","metadata":{"id":"ZOcWwlvxQoSJ"}},{"cell_type":"markdown","source":"### Download the Data to Colab\n\nUpload your `kaggle.json` to Colab. Get it here: https://www.kaggle.com/docs/api#authentication\n\n","metadata":{"id":"xCBkYxdzRPPq"}},{"cell_type":"code","source":"!ls .","metadata":{"executionInfo":{"elapsed":1627,"status":"ok","timestamp":1667911859298,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"Y-gnbjWgRg13","outputId":"c2277c8e-c5b9-4ebf-ac12-76939826bc07","execution":{"iopub.status.busy":"2023-03-01T06:12:26.984224Z","iopub.execute_input":"2023-03-01T06:12:26.984630Z","iopub.status.idle":"2023-03-01T06:12:28.124720Z","shell.execute_reply.started":"2023-03-01T06:12:26.984593Z","shell.execute_reply":"2023-03-01T06:12:28.123364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os","metadata":{"executionInfo":{"elapsed":1,"status":"ok","timestamp":1667911861246,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ADfR5g8MSZ0_","execution":{"iopub.status.busy":"2023-03-01T06:12:28.127015Z","iopub.execute_input":"2023-03-01T06:12:28.127601Z","iopub.status.idle":"2023-03-01T06:12:28.133066Z","shell.execute_reply.started":"2023-03-01T06:12:28.127555Z","shell.execute_reply":"2023-03-01T06:12:28.131785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"IS_KAGGLE = 'KAGGLE_KERNEL_RUN_TYPE' in os.environ","metadata":{"executionInfo":{"elapsed":371,"status":"ok","timestamp":1667918186535,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ixLiVhlOq7pr","execution":{"iopub.status.busy":"2023-03-01T06:12:28.134491Z","iopub.execute_input":"2023-03-01T06:12:28.134888Z","iopub.status.idle":"2023-03-01T06:12:28.144375Z","shell.execute_reply.started":"2023-03-01T06:12:28.134839Z","shell.execute_reply":"2023-03-01T06:12:28.143443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if IS_KAGGLE:\n    data_dir = '../input/quora-insincere-questions-classification'\n    train_fname = data_dir + '/train.csv'\n    test_fname = data_dir + '/test.csv'\n    sample_fname = data_dir + '/sample_submission.csv'\nelse:\n    os.environ['KAGGLE_CONFIG_DIR'] = '.'\n    !kaggle competitions download -c quora-insincere-questions-classification -f train.csv -p data\n    !kaggle competitions download -c quora-insincere-questions-classification -f test.csv -p data\n    !kaggle competitions download -c quora-insincere-questions-classification -f sample_submission.csv -p data\n    train_fname = 'data/train.csv.zip'\n    test_fname = 'data/test.csv.zip'\n    sample_fname = 'data/sample_submission.csv.zip' ","metadata":{"executionInfo":{"elapsed":3782,"status":"ok","timestamp":1667918196521,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"Wizvk8maq546","outputId":"bb8ced58-eb20-4430-db02-6d09e4b2f1d6","execution":{"iopub.status.busy":"2023-03-01T06:12:28.147561Z","iopub.execute_input":"2023-03-01T06:12:28.148650Z","iopub.status.idle":"2023-03-01T06:12:28.160931Z","shell.execute_reply.started":"2023-03-01T06:12:28.148601Z","shell.execute_reply":"2023-03-01T06:12:28.160047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Explore the Data using Pandas","metadata":{"id":"Ql6Gv4shTSBJ"}},{"cell_type":"code","source":"import pandas as pd","metadata":{"executionInfo":{"elapsed":358,"status":"ok","timestamp":1667912043319,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"pf65NRzdQ-hU","execution":{"iopub.status.busy":"2023-03-01T06:12:28.162343Z","iopub.execute_input":"2023-03-01T06:12:28.163147Z","iopub.status.idle":"2023-03-01T06:12:28.175850Z","shell.execute_reply.started":"2023-03-01T06:12:28.163108Z","shell.execute_reply":"2023-03-01T06:12:28.174564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"raw_df = pd.read_csv(train_fname)","metadata":{"executionInfo":{"elapsed":4754,"status":"ok","timestamp":1667912065510,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"H1FjN5gDTfUY","execution":{"iopub.status.busy":"2023-03-01T06:12:28.177205Z","iopub.execute_input":"2023-03-01T06:12:28.178174Z","iopub.status.idle":"2023-03-01T06:12:31.198598Z","shell.execute_reply.started":"2023-03-01T06:12:28.178127Z","shell.execute_reply":"2023-03-01T06:12:31.197671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"raw_df","metadata":{"executionInfo":{"elapsed":474,"status":"ok","timestamp":1667912069885,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"CHMgt_5iTfRt","outputId":"7ad71a7a-b478-4a42-bce0-ee467479480f","execution":{"iopub.status.busy":"2023-03-01T06:12:31.200161Z","iopub.execute_input":"2023-03-01T06:12:31.201233Z","iopub.status.idle":"2023-03-01T06:12:31.217708Z","shell.execute_reply.started":"2023-03-01T06:12:31.201181Z","shell.execute_reply":"2023-03-01T06:12:31.216073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sincere_df = raw_df[raw_df.target == 0]","metadata":{"executionInfo":{"elapsed":382,"status":"ok","timestamp":1667912255712,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"wV2xGZ3STfPx","execution":{"iopub.status.busy":"2023-03-01T06:12:31.219993Z","iopub.execute_input":"2023-03-01T06:12:31.220779Z","iopub.status.idle":"2023-03-01T06:12:31.303166Z","shell.execute_reply.started":"2023-03-01T06:12:31.220725Z","shell.execute_reply":"2023-03-01T06:12:31.302203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sincere_df.question_text.values[:10]","metadata":{"executionInfo":{"elapsed":375,"status":"ok","timestamp":1667912261902,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"me4srOFdUMCQ","outputId":"6f5e9340-bdb2-4c22-b75e-57177ccf6ea9","execution":{"iopub.status.busy":"2023-03-01T06:12:31.304392Z","iopub.execute_input":"2023-03-01T06:12:31.304710Z","iopub.status.idle":"2023-03-01T06:12:31.312097Z","shell.execute_reply.started":"2023-03-01T06:12:31.304672Z","shell.execute_reply":"2023-03-01T06:12:31.311271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"insincere_df = raw_df[raw_df.target == 1]","metadata":{"executionInfo":{"elapsed":572,"status":"ok","timestamp":1667912293701,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"cTApBtxIUMAJ","execution":{"iopub.status.busy":"2023-03-01T06:12:31.315697Z","iopub.execute_input":"2023-03-01T06:12:31.316967Z","iopub.status.idle":"2023-03-01T06:12:31.341483Z","shell.execute_reply.started":"2023-03-01T06:12:31.316916Z","shell.execute_reply":"2023-03-01T06:12:31.340339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"insincere_df.question_text.values[:10]","metadata":{"executionInfo":{"elapsed":652,"status":"ok","timestamp":1667912301181,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"-nFN5bZVUL91","outputId":"ce551e6c-3255-408d-9182-4408269d67bf","execution":{"iopub.status.busy":"2023-03-01T06:12:31.342918Z","iopub.execute_input":"2023-03-01T06:12:31.343246Z","iopub.status.idle":"2023-03-01T06:12:31.351389Z","shell.execute_reply.started":"2023-03-01T06:12:31.343216Z","shell.execute_reply":"2023-03-01T06:12:31.350570Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"raw_df.target.value_counts(normalize=True)","metadata":{"executionInfo":{"elapsed":431,"status":"ok","timestamp":1667912372012,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"n2VQSfKmUn7Z","outputId":"abe27eed-82e0-472d-f0bb-3c28a5a77b06","execution":{"iopub.status.busy":"2023-03-01T06:12:31.352727Z","iopub.execute_input":"2023-03-01T06:12:31.353840Z","iopub.status.idle":"2023-03-01T06:12:31.375759Z","shell.execute_reply.started":"2023-03-01T06:12:31.353805Z","shell.execute_reply":"2023-03-01T06:12:31.374514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"raw_df.target.value_counts(normalize=True).plot(kind='bar')","metadata":{"executionInfo":{"elapsed":395,"status":"ok","timestamp":1667912455099,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ZSzHylWuU0MF","outputId":"557af698-754b-49e6-a8f7-a504aaa74737","execution":{"iopub.status.busy":"2023-03-01T06:12:31.377362Z","iopub.execute_input":"2023-03-01T06:12:31.377718Z","iopub.status.idle":"2023-03-01T06:12:31.586612Z","shell.execute_reply.started":"2023-03-01T06:12:31.377686Z","shell.execute_reply":"2023-03-01T06:12:31.585152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = pd.read_csv(test_fname)","metadata":{"executionInfo":{"elapsed":1646,"status":"ok","timestamp":1667912505443,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"_lA6Vb6YVN1p","execution":{"iopub.status.busy":"2023-03-01T06:12:31.588406Z","iopub.execute_input":"2023-03-01T06:12:31.588821Z","iopub.status.idle":"2023-03-01T06:12:32.454180Z","shell.execute_reply.started":"2023-03-01T06:12:31.588775Z","shell.execute_reply":"2023-03-01T06:12:32.453147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df","metadata":{"executionInfo":{"elapsed":3,"status":"ok","timestamp":1667912507665,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"gH0ij_pmVNzV","outputId":"ad30dad3-ffd4-43ac-c04d-659ac7cf006d","execution":{"iopub.status.busy":"2023-03-01T06:12:32.455516Z","iopub.execute_input":"2023-03-01T06:12:32.455844Z","iopub.status.idle":"2023-03-01T06:12:32.469113Z","shell.execute_reply.started":"2023-03-01T06:12:32.455813Z","shell.execute_reply":"2023-03-01T06:12:32.467953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df = pd.read_csv(sample_fname)","metadata":{"executionInfo":{"elapsed":532,"status":"ok","timestamp":1667912554978,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"exDbcT9bVNw0","execution":{"iopub.status.busy":"2023-03-01T06:12:32.471644Z","iopub.execute_input":"2023-03-01T06:12:32.472096Z","iopub.status.idle":"2023-03-01T06:12:32.735558Z","shell.execute_reply.started":"2023-03-01T06:12:32.472058Z","shell.execute_reply":"2023-03-01T06:12:32.734485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df","metadata":{"executionInfo":{"elapsed":571,"status":"ok","timestamp":1667912558749,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"vLCvMxN3UL7l","outputId":"75dcfd90-504e-4612-b578-dcff4978dc00","execution":{"iopub.status.busy":"2023-03-01T06:12:32.736824Z","iopub.execute_input":"2023-03-01T06:12:32.737191Z","iopub.status.idle":"2023-03-01T06:12:32.750611Z","shell.execute_reply.started":"2023-03-01T06:12:32.737157Z","shell.execute_reply":"2023-03-01T06:12:32.749391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df.prediction.value_counts()","metadata":{"executionInfo":{"elapsed":394,"status":"ok","timestamp":1667912639872,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"BvL2NJuSVv4l","outputId":"d362dd88-f047-4ca6-e729-54b153126cb1","execution":{"iopub.status.busy":"2023-03-01T06:12:32.752461Z","iopub.execute_input":"2023-03-01T06:12:32.753273Z","iopub.status.idle":"2023-03-01T06:12:32.767479Z","shell.execute_reply.started":"2023-03-01T06:12:32.753218Z","shell.execute_reply":"2023-03-01T06:12:32.766277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create a Working Sample\n","metadata":{"id":"v7sWG0KCV0g-"}},{"cell_type":"code","source":"if IS_KAGGLE:\n    SAMPLE_SIZE = len(raw_df)\nelse:\n    SAMPLE_SIZE = 100_000","metadata":{"executionInfo":{"elapsed":653,"status":"ok","timestamp":1667912690959,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"hu5s7hLKV7Kr","execution":{"iopub.status.busy":"2023-03-01T06:12:32.768847Z","iopub.execute_input":"2023-03-01T06:12:32.772349Z","iopub.status.idle":"2023-03-01T06:12:32.777958Z","shell.execute_reply.started":"2023-03-01T06:12:32.772277Z","shell.execute_reply":"2023-03-01T06:12:32.776947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_df = raw_df.sample(SAMPLE_SIZE, random_state=42)","metadata":{"executionInfo":{"elapsed":702,"status":"ok","timestamp":1667912713848,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"0xzqyYo_V31J","execution":{"iopub.status.busy":"2023-03-01T06:12:32.779368Z","iopub.execute_input":"2023-03-01T06:12:32.779680Z","iopub.status.idle":"2023-03-01T06:12:33.223712Z","shell.execute_reply.started":"2023-03-01T06:12:32.779650Z","shell.execute_reply":"2023-03-01T06:12:33.222508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_df","metadata":{"executionInfo":{"elapsed":886,"status":"ok","timestamp":1667912722831,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"4xFSOIfdV3yh","outputId":"a1f8d588-1005-475a-bfa9-04df72110179","execution":{"iopub.status.busy":"2023-03-01T06:12:33.224946Z","iopub.execute_input":"2023-03-01T06:12:33.225311Z","iopub.status.idle":"2023-03-01T06:12:33.238869Z","shell.execute_reply.started":"2023-03-01T06:12:33.225277Z","shell.execute_reply":"2023-03-01T06:12:33.237561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Text Preprocessing Techniques\n\nOutline:\n\n1. Understand the bag of words model\n2. Tokenization\n3. Stop word removal\n4. Stemming","metadata":{"id":"-gZy068pQoPg"}},{"cell_type":"markdown","source":"### Bag of Words Intuition\n\n1. Create a list of all the words across all the text documents\n2. You convert each document into vector counts of each word\n\n\nLimitations:\n1. There may be too many words in the dataset\n2. Some words may occur too frequently\n3. Some words may occur very rarely or only once\n4. A single word may have many forms (go, gone, going or bird vs. birds)","metadata":{"id":"pJ0mh3gjXH9e"}},{"cell_type":"code","source":"q0 = sincere_df.question_text.values[1]","metadata":{"executionInfo":{"elapsed":628,"status":"ok","timestamp":1667913361214,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"PDb6D4pfYWPu","execution":{"iopub.status.busy":"2023-03-01T06:12:33.240248Z","iopub.execute_input":"2023-03-01T06:12:33.241249Z","iopub.status.idle":"2023-03-01T06:12:33.245639Z","shell.execute_reply.started":"2023-03-01T06:12:33.241213Z","shell.execute_reply":"2023-03-01T06:12:33.244513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0","metadata":{"executionInfo":{"elapsed":546,"status":"ok","timestamp":1667913367151,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"mQjFxTKFYjJk","outputId":"3012387d-9827-46bf-a3bb-d9d077ee6a9f","execution":{"iopub.status.busy":"2023-03-01T06:12:33.248847Z","iopub.execute_input":"2023-03-01T06:12:33.249900Z","iopub.status.idle":"2023-03-01T06:12:33.258753Z","shell.execute_reply.started":"2023-03-01T06:12:33.249861Z","shell.execute_reply":"2023-03-01T06:12:33.257677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1 = raw_df[raw_df.target == 1].question_text.values[0]","metadata":{"executionInfo":{"elapsed":478,"status":"ok","timestamp":1667913449142,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"GIvt_M-yYlJH","execution":{"iopub.status.busy":"2023-03-01T06:12:33.260234Z","iopub.execute_input":"2023-03-01T06:12:33.260592Z","iopub.status.idle":"2023-03-01T06:12:33.290535Z","shell.execute_reply.started":"2023-03-01T06:12:33.260546Z","shell.execute_reply":"2023-03-01T06:12:33.289501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1","metadata":{"executionInfo":{"elapsed":3,"status":"ok","timestamp":1667913449769,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"86bHxTMbY182","outputId":"ffafd417-1607-420d-a2ff-9169707470b2","execution":{"iopub.status.busy":"2023-03-01T06:12:33.291759Z","iopub.execute_input":"2023-03-01T06:12:33.292087Z","iopub.status.idle":"2023-03-01T06:12:33.300223Z","shell.execute_reply.started":"2023-03-01T06:12:33.292056Z","shell.execute_reply":"2023-03-01T06:12:33.299040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Tokenization\n\nsplitting a document into words and separators","metadata":{"id":"qj4Zg1yaYPJv"}},{"cell_type":"code","source":"import nltk","metadata":{"executionInfo":{"elapsed":474,"status":"ok","timestamp":1667913549193,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"EqacxiuyZPds","execution":{"iopub.status.busy":"2023-03-01T06:12:33.301926Z","iopub.execute_input":"2023-03-01T06:12:33.302338Z","iopub.status.idle":"2023-03-01T06:12:33.308856Z","shell.execute_reply.started":"2023-03-01T06:12:33.302303Z","shell.execute_reply":"2023-03-01T06:12:33.307503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from nltk.tokenize import word_tokenize","metadata":{"executionInfo":{"elapsed":1852,"status":"ok","timestamp":1667913505183,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"2vQDMEq0Q_lP","execution":{"iopub.status.busy":"2023-03-01T06:12:33.312175Z","iopub.execute_input":"2023-03-01T06:12:33.313268Z","iopub.status.idle":"2023-03-01T06:12:33.319424Z","shell.execute_reply.started":"2023-03-01T06:12:33.313219Z","shell.execute_reply":"2023-03-01T06:12:33.318481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nltk.download('punkt')","metadata":{"executionInfo":{"elapsed":639,"status":"ok","timestamp":1667913551680,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"5eaAldKAZN-4","outputId":"d419ecd2-0540-4ed5-f0fc-32e68bd287dc","execution":{"iopub.status.busy":"2023-03-01T06:12:33.329338Z","iopub.execute_input":"2023-03-01T06:12:33.329690Z","iopub.status.idle":"2023-03-01T06:12:33.338037Z","shell.execute_reply.started":"2023-03-01T06:12:33.329657Z","shell.execute_reply":"2023-03-01T06:12:33.336782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0","metadata":{"executionInfo":{"elapsed":1019,"status":"ok","timestamp":1667913515768,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"VyIaEATyQ_ip","outputId":"655a30fa-ccf0-4839-aa2a-15f4437a5562","execution":{"iopub.status.busy":"2023-03-01T06:12:33.339681Z","iopub.execute_input":"2023-03-01T06:12:33.340122Z","iopub.status.idle":"2023-03-01T06:12:33.348002Z","shell.execute_reply.started":"2023-03-01T06:12:33.340077Z","shell.execute_reply":"2023-03-01T06:12:33.346750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"word_tokenize(q0)","metadata":{"executionInfo":{"elapsed":458,"status":"ok","timestamp":1667913557740,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"1yULG5nVQ_gk","outputId":"83e3941f-3d29-4008-c885-b9a5eaf93912","execution":{"iopub.status.busy":"2023-03-01T06:12:33.349420Z","iopub.execute_input":"2023-03-01T06:12:33.349841Z","iopub.status.idle":"2023-03-01T06:12:33.360337Z","shell.execute_reply.started":"2023-03-01T06:12:33.349795Z","shell.execute_reply":"2023-03-01T06:12:33.359520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"word_tokenize(' this is (something) with, a lot of, punctuation;')","metadata":{"executionInfo":{"elapsed":451,"status":"ok","timestamp":1667913596244,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"gJ_lrgEpZIt8","outputId":"aa250716-93b6-47ab-cbd1-d9ba9076c620","execution":{"iopub.status.busy":"2023-03-01T06:12:33.361539Z","iopub.execute_input":"2023-03-01T06:12:33.361845Z","iopub.status.idle":"2023-03-01T06:12:33.375862Z","shell.execute_reply.started":"2023-03-01T06:12:33.361816Z","shell.execute_reply":"2023-03-01T06:12:33.374509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1","metadata":{"executionInfo":{"elapsed":913,"status":"ok","timestamp":1667913614596,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"uoP8H4KcZIr1","outputId":"0800baed-d874-4f5c-ba0b-fb5aad5b0fc0","execution":{"iopub.status.busy":"2023-03-01T06:12:33.377618Z","iopub.execute_input":"2023-03-01T06:12:33.378036Z","iopub.status.idle":"2023-03-01T06:12:33.384875Z","shell.execute_reply.started":"2023-03-01T06:12:33.377994Z","shell.execute_reply":"2023-03-01T06:12:33.383949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"word_tokenize(q1)","metadata":{"executionInfo":{"elapsed":977,"status":"ok","timestamp":1667913623231,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"Bs_0YAawZIpf","outputId":"e97ea6f2-c22c-4f06-fe1f-160df99340ad","execution":{"iopub.status.busy":"2023-03-01T06:12:33.386372Z","iopub.execute_input":"2023-03-01T06:12:33.386692Z","iopub.status.idle":"2023-03-01T06:12:33.397740Z","shell.execute_reply.started":"2023-03-01T06:12:33.386662Z","shell.execute_reply":"2023-03-01T06:12:33.396598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_tok = word_tokenize(q0)\nq1_tok = word_tokenize(q1)","metadata":{"executionInfo":{"elapsed":1,"status":"ok","timestamp":1667913656571,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"7HNGqFYtZInd","execution":{"iopub.status.busy":"2023-03-01T06:12:33.399389Z","iopub.execute_input":"2023-03-01T06:12:33.400181Z","iopub.status.idle":"2023-03-01T06:12:33.407891Z","shell.execute_reply.started":"2023-03-01T06:12:33.400143Z","shell.execute_reply":"2023-03-01T06:12:33.406819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Stop Word Removal\n\nRemoving commonly occuring words","metadata":{"id":"Mf5aqvkGZrR-"}},{"cell_type":"code","source":"q1_tok","metadata":{"executionInfo":{"elapsed":385,"status":"ok","timestamp":1667913691780,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"7yZhlXV9Zxo-","outputId":"408a649d-3883-4f70-e1b5-a052ada07d60","execution":{"iopub.status.busy":"2023-03-01T06:12:33.409475Z","iopub.execute_input":"2023-03-01T06:12:33.409811Z","iopub.status.idle":"2023-03-01T06:12:33.422047Z","shell.execute_reply.started":"2023-03-01T06:12:33.409780Z","shell.execute_reply":"2023-03-01T06:12:33.420661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from nltk.corpus import stopwords","metadata":{"executionInfo":{"elapsed":1009,"status":"ok","timestamp":1667913739715,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"s1qsENBhZrD0","execution":{"iopub.status.busy":"2023-03-01T06:12:33.423484Z","iopub.execute_input":"2023-03-01T06:12:33.423818Z","iopub.status.idle":"2023-03-01T06:12:33.431037Z","shell.execute_reply.started":"2023-03-01T06:12:33.423778Z","shell.execute_reply":"2023-03-01T06:12:33.430131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nltk.download('stopwords')","metadata":{"executionInfo":{"elapsed":459,"status":"ok","timestamp":1667913791292,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"4kjpGIjraK7r","outputId":"dc16bc42-9e3e-4796-8937-5d9af34d099e","execution":{"iopub.status.busy":"2023-03-01T06:12:33.432200Z","iopub.execute_input":"2023-03-01T06:12:33.433018Z","iopub.status.idle":"2023-03-01T06:12:33.446514Z","shell.execute_reply.started":"2023-03-01T06:12:33.432971Z","shell.execute_reply":"2023-03-01T06:12:33.445348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"english_stopwords = stopwords.words('english')","metadata":{"executionInfo":{"elapsed":430,"status":"ok","timestamp":1667913795278,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ejhkI9oVZrCD","execution":{"iopub.status.busy":"2023-03-01T06:12:33.448256Z","iopub.execute_input":"2023-03-01T06:12:33.448708Z","iopub.status.idle":"2023-03-01T06:12:33.456729Z","shell.execute_reply.started":"2023-03-01T06:12:33.448661Z","shell.execute_reply":"2023-03-01T06:12:33.455765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\", \".join(english_stopwords)","metadata":{"executionInfo":{"elapsed":357,"status":"ok","timestamp":1667913817359,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"jB5avxBsZqxB","outputId":"27c5b385-e8e1-4987-cc2a-19d43eb70308","execution":{"iopub.status.busy":"2023-03-01T06:12:33.458119Z","iopub.execute_input":"2023-03-01T06:12:33.459257Z","iopub.status.idle":"2023-03-01T06:12:33.468778Z","shell.execute_reply.started":"2023-03-01T06:12:33.459203Z","shell.execute_reply":"2023-03-01T06:12:33.467683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def remove_stopwords(tokens):\n    return [word for word in tokens if word.lower() not in english_stopwords]","metadata":{"executionInfo":{"elapsed":352,"status":"ok","timestamp":1667913917513,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"7umWfNCvZqsF","execution":{"iopub.status.busy":"2023-03-01T06:12:33.470505Z","iopub.execute_input":"2023-03-01T06:12:33.471319Z","iopub.status.idle":"2023-03-01T06:12:33.478891Z","shell.execute_reply.started":"2023-03-01T06:12:33.471283Z","shell.execute_reply":"2023-03-01T06:12:33.477923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_tok","metadata":{"executionInfo":{"elapsed":930,"status":"ok","timestamp":1667913931371,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"15NSoE_Saq6W","outputId":"4abcb5d0-fae7-4c59-f01d-c8d641d798e9","execution":{"iopub.status.busy":"2023-03-01T06:12:33.480346Z","iopub.execute_input":"2023-03-01T06:12:33.481314Z","iopub.status.idle":"2023-03-01T06:12:33.495344Z","shell.execute_reply.started":"2023-03-01T06:12:33.481280Z","shell.execute_reply":"2023-03-01T06:12:33.494110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_stp = remove_stopwords(q0_tok)","metadata":{"executionInfo":{"elapsed":667,"status":"ok","timestamp":1667913944997,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"b59BPAClaq4B","execution":{"iopub.status.busy":"2023-03-01T06:12:33.496754Z","iopub.execute_input":"2023-03-01T06:12:33.497126Z","iopub.status.idle":"2023-03-01T06:12:33.504286Z","shell.execute_reply.started":"2023-03-01T06:12:33.497092Z","shell.execute_reply":"2023-03-01T06:12:33.503380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_stp","metadata":{"executionInfo":{"elapsed":502,"status":"ok","timestamp":1667913950071,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"UJpOKfd-Zqp_","outputId":"d9cd0cfc-26a5-45a2-eb18-3517d13bedda","execution":{"iopub.status.busy":"2023-03-01T06:12:33.505465Z","iopub.execute_input":"2023-03-01T06:12:33.506433Z","iopub.status.idle":"2023-03-01T06:12:33.519250Z","shell.execute_reply.started":"2023-03-01T06:12:33.506385Z","shell.execute_reply":"2023-03-01T06:12:33.518141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_stp = remove_stopwords(q1_tok)","metadata":{"executionInfo":{"elapsed":476,"status":"ok","timestamp":1667913994939,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"IKR9dyGVa6OR","execution":{"iopub.status.busy":"2023-03-01T06:12:33.521007Z","iopub.execute_input":"2023-03-01T06:12:33.522266Z","iopub.status.idle":"2023-03-01T06:12:33.530032Z","shell.execute_reply.started":"2023-03-01T06:12:33.522217Z","shell.execute_reply":"2023-03-01T06:12:33.528921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_tok","metadata":{"executionInfo":{"elapsed":699,"status":"ok","timestamp":1667913999121,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"_RvMiZAka6Lv","outputId":"4f446d13-cf60-4da4-e6d6-50b9f46a1ad0","execution":{"iopub.status.busy":"2023-03-01T06:12:33.531764Z","iopub.execute_input":"2023-03-01T06:12:33.532704Z","iopub.status.idle":"2023-03-01T06:12:33.542949Z","shell.execute_reply.started":"2023-03-01T06:12:33.532656Z","shell.execute_reply":"2023-03-01T06:12:33.541947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_stp","metadata":{"executionInfo":{"elapsed":485,"status":"ok","timestamp":1667914001968,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"GqpqLKw9a-A8","outputId":"9280f918-57dc-4673-bb3b-274107f8b62b","execution":{"iopub.status.busy":"2023-03-01T06:12:33.544279Z","iopub.execute_input":"2023-03-01T06:12:33.545285Z","iopub.status.idle":"2023-03-01T06:12:33.553036Z","shell.execute_reply.started":"2023-03-01T06:12:33.545249Z","shell.execute_reply":"2023-03-01T06:12:33.552037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Stemming\n\n\"go\", \"gone\", \"going\" -> \"go\"\n\"birds\", \"bird\" -> \"bird\"","metadata":{"id":"FeVx68x5bHEF"}},{"cell_type":"code","source":"from nltk.stem.snowball import SnowballStemmer","metadata":{"executionInfo":{"elapsed":1054,"status":"ok","timestamp":1667914103321,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"9jFOfSqtbG6y","execution":{"iopub.status.busy":"2023-03-01T06:12:33.554153Z","iopub.execute_input":"2023-03-01T06:12:33.555137Z","iopub.status.idle":"2023-03-01T06:12:33.562556Z","shell.execute_reply.started":"2023-03-01T06:12:33.554956Z","shell.execute_reply":"2023-03-01T06:12:33.561465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stemmer = SnowballStemmer(language='english')","metadata":{"executionInfo":{"elapsed":544,"status":"ok","timestamp":1667914116697,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"SC0PMIy3bG4r","execution":{"iopub.status.busy":"2023-03-01T06:12:33.564448Z","iopub.execute_input":"2023-03-01T06:12:33.564904Z","iopub.status.idle":"2023-03-01T06:12:33.573252Z","shell.execute_reply.started":"2023-03-01T06:12:33.564858Z","shell.execute_reply":"2023-03-01T06:12:33.572019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stemmer.stem('going')","metadata":{"executionInfo":{"elapsed":792,"status":"ok","timestamp":1667914135778,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"apl0nhFqbG2F","outputId":"fc70d0f5-0f7a-49b4-ace3-ea1113cb8c66","execution":{"iopub.status.busy":"2023-03-01T06:12:33.574886Z","iopub.execute_input":"2023-03-01T06:12:33.575357Z","iopub.status.idle":"2023-03-01T06:12:33.586547Z","shell.execute_reply.started":"2023-03-01T06:12:33.575308Z","shell.execute_reply":"2023-03-01T06:12:33.585275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stemmer.stem('supposedly')","metadata":{"executionInfo":{"elapsed":386,"status":"ok","timestamp":1667914142907,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"wtIoHT1ybGz2","outputId":"3609cf3d-2c9f-4638-db1f-07ce31e63f20","execution":{"iopub.status.busy":"2023-03-01T06:12:33.588163Z","iopub.execute_input":"2023-03-01T06:12:33.589253Z","iopub.status.idle":"2023-03-01T06:12:33.598957Z","shell.execute_reply.started":"2023-03-01T06:12:33.589194Z","shell.execute_reply":"2023-03-01T06:12:33.597831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_stm = [stemmer.stem(word) for word in q0_stp]","metadata":{"executionInfo":{"elapsed":364,"status":"ok","timestamp":1667914180406,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"hGdECu-rbmbZ","execution":{"iopub.status.busy":"2023-03-01T06:12:33.600415Z","iopub.execute_input":"2023-03-01T06:12:33.601279Z","iopub.status.idle":"2023-03-01T06:12:33.610566Z","shell.execute_reply.started":"2023-03-01T06:12:33.601242Z","shell.execute_reply":"2023-03-01T06:12:33.609630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_stp","metadata":{"executionInfo":{"elapsed":817,"status":"ok","timestamp":1667914186316,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"BY0fs95pbqtH","outputId":"5e0d4927-385f-40c7-ad88-82ec30049890","execution":{"iopub.status.busy":"2023-03-01T06:12:33.611904Z","iopub.execute_input":"2023-03-01T06:12:33.612385Z","iopub.status.idle":"2023-03-01T06:12:33.623611Z","shell.execute_reply.started":"2023-03-01T06:12:33.612350Z","shell.execute_reply":"2023-03-01T06:12:33.622438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q0_stm","metadata":{"executionInfo":{"elapsed":573,"status":"ok","timestamp":1667914193742,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ARuFpGkHbqn2","outputId":"3e4e85e5-bb93-4feb-97f2-5141ebbcd1db","execution":{"iopub.status.busy":"2023-03-01T06:12:33.625172Z","iopub.execute_input":"2023-03-01T06:12:33.626091Z","iopub.status.idle":"2023-03-01T06:12:33.635125Z","shell.execute_reply.started":"2023-03-01T06:12:33.625927Z","shell.execute_reply":"2023-03-01T06:12:33.634055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_stm = [stemmer.stem(word) for word in q1_stp]","metadata":{"executionInfo":{"elapsed":432,"status":"ok","timestamp":1667914233752,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"kiFiGdCpbrot","execution":{"iopub.status.busy":"2023-03-01T06:12:33.636754Z","iopub.execute_input":"2023-03-01T06:12:33.637864Z","iopub.status.idle":"2023-03-01T06:12:33.643213Z","shell.execute_reply.started":"2023-03-01T06:12:33.637827Z","shell.execute_reply":"2023-03-01T06:12:33.642235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_stp","metadata":{"executionInfo":{"elapsed":458,"status":"ok","timestamp":1667914238715,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"WweHQFO4brmn","outputId":"58cc1dae-68c0-4778-9fb1-27e37e054e86","execution":{"iopub.status.busy":"2023-03-01T06:12:33.644823Z","iopub.execute_input":"2023-03-01T06:12:33.645448Z","iopub.status.idle":"2023-03-01T06:12:33.656557Z","shell.execute_reply.started":"2023-03-01T06:12:33.645413Z","shell.execute_reply":"2023-03-01T06:12:33.655277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q1_stm","metadata":{"executionInfo":{"elapsed":906,"status":"ok","timestamp":1667914244006,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"AVu5FiAbbrkV","outputId":"4a4c2193-e5b7-4f1c-8781-127e2f84efb0","execution":{"iopub.status.busy":"2023-03-01T06:12:33.657969Z","iopub.execute_input":"2023-03-01T06:12:33.658317Z","iopub.status.idle":"2023-03-01T06:12:33.668504Z","shell.execute_reply.started":"2023-03-01T06:12:33.658286Z","shell.execute_reply":"2023-03-01T06:12:33.667331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Lemmatization\n\n\"love\" -> \"love\"\n\"loving\" -> \"love\"\n\"lovable\" -> \"love\"\n","metadata":{"id":"VWWkiTcWcAR7"}},{"cell_type":"markdown","source":"## Implement Bag of Words \n\n\nOutline:\n\n1. Create a vocabulary using Count Vectorizer\n2. Transform text to vectors using Count Vectorizer\n3. Configure text preprocessing in Count Vectorizer","metadata":{"id":"CuTR9sOWQoMq"}},{"cell_type":"markdown","source":"### Create a Vocabulary","metadata":{"id":"0HWShc6jdPFy"}},{"cell_type":"code","source":"sample_df","metadata":{"executionInfo":{"elapsed":419,"status":"ok","timestamp":1667914532328,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"uGScEAd9c_dQ","outputId":"e923813b-b8fd-438b-a6f0-743c280a204d","execution":{"iopub.status.busy":"2023-03-01T06:12:33.670189Z","iopub.execute_input":"2023-03-01T06:12:33.670928Z","iopub.status.idle":"2023-03-01T06:12:33.687055Z","shell.execute_reply.started":"2023-03-01T06:12:33.670889Z","shell.execute_reply":"2023-03-01T06:12:33.685701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_df = sample_df[:5]","metadata":{"executionInfo":{"elapsed":707,"status":"ok","timestamp":1667914559612,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"IKj0BM2NRATm","execution":{"iopub.status.busy":"2023-03-01T06:12:33.688435Z","iopub.execute_input":"2023-03-01T06:12:33.688739Z","iopub.status.idle":"2023-03-01T06:12:33.694147Z","shell.execute_reply.started":"2023-03-01T06:12:33.688710Z","shell.execute_reply":"2023-03-01T06:12:33.692854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_df","metadata":{"executionInfo":{"elapsed":1222,"status":"ok","timestamp":1667914576774,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"awqkIOZEdKFu","outputId":"9c110405-71e1-4c72-c02c-dfbc1d5be43c","execution":{"iopub.status.busy":"2023-03-01T06:12:33.695863Z","iopub.execute_input":"2023-03-01T06:12:33.697105Z","iopub.status.idle":"2023-03-01T06:12:33.711830Z","shell.execute_reply.started":"2023-03-01T06:12:33.697052Z","shell.execute_reply":"2023-03-01T06:12:33.710876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_df.question_text.values","metadata":{"executionInfo":{"elapsed":430,"status":"ok","timestamp":1667914566540,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"XZOIXmC3RARh","outputId":"330c3a06-1368-40de-ae9c-40f0061e61d6","execution":{"iopub.status.busy":"2023-03-01T06:12:33.713382Z","iopub.execute_input":"2023-03-01T06:12:33.714374Z","iopub.status.idle":"2023-03-01T06:12:33.726112Z","shell.execute_reply.started":"2023-03-01T06:12:33.714333Z","shell.execute_reply":"2023-03-01T06:12:33.724944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer","metadata":{"executionInfo":{"elapsed":362,"status":"ok","timestamp":1667914669258,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"Yr3XgzBWdNNx","execution":{"iopub.status.busy":"2023-03-01T06:12:33.727568Z","iopub.execute_input":"2023-03-01T06:12:33.728178Z","iopub.status.idle":"2023-03-01T06:12:33.734931Z","shell.execute_reply.started":"2023-03-01T06:12:33.728142Z","shell.execute_reply":"2023-03-01T06:12:33.733732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_vect = CountVectorizer()","metadata":{"executionInfo":{"elapsed":351,"status":"ok","timestamp":1667914677547,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"-sE8WL3-dNLf","execution":{"iopub.status.busy":"2023-03-01T06:12:33.736591Z","iopub.execute_input":"2023-03-01T06:12:33.737684Z","iopub.status.idle":"2023-03-01T06:12:33.745094Z","shell.execute_reply.started":"2023-03-01T06:12:33.737636Z","shell.execute_reply":"2023-03-01T06:12:33.744020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_vect.fit(small_df.question_text)","metadata":{"executionInfo":{"elapsed":409,"status":"ok","timestamp":1667914713345,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ba9oOkD6RAPP","outputId":"eccc7e9b-266b-4aea-90fa-654fa176b504","execution":{"iopub.status.busy":"2023-03-01T06:12:33.746620Z","iopub.execute_input":"2023-03-01T06:12:33.747735Z","iopub.status.idle":"2023-03-01T06:12:33.760284Z","shell.execute_reply.started":"2023-03-01T06:12:33.747684Z","shell.execute_reply":"2023-03-01T06:12:33.758768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_vect.get_feature_names_out()","metadata":{"executionInfo":{"elapsed":651,"status":"ok","timestamp":1667914761316,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"-QPGkxLZdkPy","outputId":"7de3797e-3a41-4bb1-c581-ad7bd7610eb4","execution":{"iopub.status.busy":"2023-03-01T06:12:33.762203Z","iopub.execute_input":"2023-03-01T06:12:33.762973Z","iopub.status.idle":"2023-03-01T06:12:33.771404Z","shell.execute_reply.started":"2023-03-01T06:12:33.762923Z","shell.execute_reply":"2023-03-01T06:12:33.770148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Transform documents into Vectors","metadata":{"id":"WXsUWz_hd9KE"}},{"cell_type":"code","source":"vectors = small_vect.transform(small_df.question_text)","metadata":{"executionInfo":{"elapsed":533,"status":"ok","timestamp":1667914847949,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"nZVZ3jIddkKT","execution":{"iopub.status.busy":"2023-03-01T06:12:33.773104Z","iopub.execute_input":"2023-03-01T06:12:33.773862Z","iopub.status.idle":"2023-03-01T06:12:33.783361Z","shell.execute_reply.started":"2023-03-01T06:12:33.773813Z","shell.execute_reply":"2023-03-01T06:12:33.782210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectors","metadata":{"executionInfo":{"elapsed":638,"status":"ok","timestamp":1667914853853,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"MIGbsQEQdkHt","outputId":"1ad2eff7-d795-4c0e-ab60-f88ce7b4ed8c","execution":{"iopub.status.busy":"2023-03-01T06:12:33.785060Z","iopub.execute_input":"2023-03-01T06:12:33.785840Z","iopub.status.idle":"2023-03-01T06:12:33.797397Z","shell.execute_reply.started":"2023-03-01T06:12:33.785792Z","shell.execute_reply":"2023-03-01T06:12:33.795995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectors.shape","metadata":{"executionInfo":{"elapsed":630,"status":"ok","timestamp":1667914900662,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"wriwC-1DeZPA","outputId":"9b7a3cba-e4e6-4023-851b-7cb6b6029f40","execution":{"iopub.status.busy":"2023-03-01T06:12:33.800348Z","iopub.execute_input":"2023-03-01T06:12:33.801240Z","iopub.status.idle":"2023-03-01T06:12:33.809541Z","shell.execute_reply.started":"2023-03-01T06:12:33.801199Z","shell.execute_reply":"2023-03-01T06:12:33.808347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"small_df.question_text.values[0]","metadata":{"executionInfo":{"elapsed":495,"status":"ok","timestamp":1667914953574,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"bXtFEWjtekqt","outputId":"a4c659e8-4c86-478e-c827-ddbacde28bd9","execution":{"iopub.status.busy":"2023-03-01T06:12:33.810788Z","iopub.execute_input":"2023-03-01T06:12:33.811330Z","iopub.status.idle":"2023-03-01T06:12:33.820460Z","shell.execute_reply.started":"2023-03-01T06:12:33.811288Z","shell.execute_reply":"2023-03-01T06:12:33.819638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectors[0].toarray()","metadata":{"executionInfo":{"elapsed":982,"status":"ok","timestamp":1667915020439,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"etQ-T7_1e0vF","outputId":"9446f31f-ae80-4fac-857f-42f9481bafed","execution":{"iopub.status.busy":"2023-03-01T06:12:33.821679Z","iopub.execute_input":"2023-03-01T06:12:33.822274Z","iopub.status.idle":"2023-03-01T06:12:33.833110Z","shell.execute_reply.started":"2023-03-01T06:12:33.822240Z","shell.execute_reply":"2023-03-01T06:12:33.832278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectors.toarray()","metadata":{"executionInfo":{"elapsed":483,"status":"ok","timestamp":1667914889375,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"hSrf1ZPndkFX","outputId":"63dd68aa-56d5-4838-9f41-9f12c165e9b2","execution":{"iopub.status.busy":"2023-03-01T06:12:33.834381Z","iopub.execute_input":"2023-03-01T06:12:33.835277Z","iopub.status.idle":"2023-03-01T06:12:33.846082Z","shell.execute_reply.started":"2023-03-01T06:12:33.835240Z","shell.execute_reply":"2023-03-01T06:12:33.845215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Configure Count Vectorizer Parameters","metadata":{"id":"vZPCOwobe485"}},{"cell_type":"code","source":"stemmer = SnowballStemmer(language='english')","metadata":{"executionInfo":{"elapsed":391,"status":"ok","timestamp":1667915246398,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"xUjZDsdqfq4u","execution":{"iopub.status.busy":"2023-03-01T06:12:33.847290Z","iopub.execute_input":"2023-03-01T06:12:33.847630Z","iopub.status.idle":"2023-03-01T06:12:33.856668Z","shell.execute_reply.started":"2023-03-01T06:12:33.847591Z","shell.execute_reply":"2023-03-01T06:12:33.855717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def tokenize(text):\n    return [stemmer.stem(word) for word in word_tokenize(text)]","metadata":{"executionInfo":{"elapsed":368,"status":"ok","timestamp":1667915256240,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"9hZzn1LBfPoM","execution":{"iopub.status.busy":"2023-03-01T06:12:33.858022Z","iopub.execute_input":"2023-03-01T06:12:33.858646Z","iopub.status.idle":"2023-03-01T06:12:33.868290Z","shell.execute_reply.started":"2023-03-01T06:12:33.858610Z","shell.execute_reply":"2023-03-01T06:12:33.867196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tokenize('What is the really (dealing) here?')","metadata":{"executionInfo":{"elapsed":358,"status":"ok","timestamp":1667915263406,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"OAausoIrfa2I","outputId":"96c1e3a3-1f5c-40d0-e05b-1c32046aff8c","execution":{"iopub.status.busy":"2023-03-01T06:12:33.869556Z","iopub.execute_input":"2023-03-01T06:12:33.869905Z","iopub.status.idle":"2023-03-01T06:12:33.883364Z","shell.execute_reply.started":"2023-03-01T06:12:33.869872Z","shell.execute_reply":"2023-03-01T06:12:33.882231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectorizer = CountVectorizer(lowercase=True, \n                             tokenizer=tokenize,\n                             stop_words=english_stopwords,\n                             max_features=1000)","metadata":{"executionInfo":{"elapsed":559,"status":"ok","timestamp":1667915360958,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"pJGAErs7e9Fh","execution":{"iopub.status.busy":"2023-03-01T06:12:33.885095Z","iopub.execute_input":"2023-03-01T06:12:33.885683Z","iopub.status.idle":"2023-03-01T06:12:33.893454Z","shell.execute_reply.started":"2023-03-01T06:12:33.885631Z","shell.execute_reply":"2023-03-01T06:12:33.892467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nvectorizer.fit(sample_df.question_text)","metadata":{"executionInfo":{"elapsed":32457,"status":"ok","timestamp":1667915423839,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"pI8nbkpAe9Dg","outputId":"e6151146-4e99-4bb5-bf70-c7336b5ed226","execution":{"iopub.status.busy":"2023-03-01T06:12:33.894739Z","iopub.execute_input":"2023-03-01T06:12:33.895674Z","iopub.status.idle":"2023-03-01T06:19:54.730419Z","shell.execute_reply.started":"2023-03-01T06:12:33.895637Z","shell.execute_reply":"2023-03-01T06:19:54.729449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(vectorizer.vocabulary_)","metadata":{"executionInfo":{"elapsed":2068,"status":"ok","timestamp":1667915449808,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"sJBlLm5Ye88Z","outputId":"e8ad80f8-cd4d-4289-fd26-163dc808e69f","execution":{"iopub.status.busy":"2023-03-01T06:19:54.731518Z","iopub.execute_input":"2023-03-01T06:19:54.732547Z","iopub.status.idle":"2023-03-01T06:19:54.740320Z","shell.execute_reply.started":"2023-03-01T06:19:54.732497Z","shell.execute_reply":"2023-03-01T06:19:54.739189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectorizer.get_feature_names_out()[:100]","metadata":{"executionInfo":{"elapsed":1059,"status":"ok","timestamp":1667915475499,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"b9K638h1gg5_","outputId":"955fe2e0-52ab-4137-9ecc-f18a64542e45","execution":{"iopub.status.busy":"2023-03-01T06:19:54.741611Z","iopub.execute_input":"2023-03-01T06:19:54.742084Z","iopub.status.idle":"2023-03-01T06:19:54.753482Z","shell.execute_reply.started":"2023-03-01T06:19:54.742041Z","shell.execute_reply":"2023-03-01T06:19:54.752342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ninputs = vectorizer.transform(sample_df.question_text)","metadata":{"executionInfo":{"elapsed":30784,"status":"ok","timestamp":1667915566857,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"n3iq65OsguTR","outputId":"35c22bcd-b6c2-4629-9e83-ad81a29456ee","execution":{"iopub.status.busy":"2023-03-01T06:19:54.755242Z","iopub.execute_input":"2023-03-01T06:19:54.755643Z","iopub.status.idle":"2023-03-01T06:27:13.989110Z","shell.execute_reply.started":"2023-03-01T06:19:54.755612Z","shell.execute_reply":"2023-03-01T06:27:13.987888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"inputs.shape","metadata":{"executionInfo":{"elapsed":661,"status":"ok","timestamp":1667915574286,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"9aEbKIC0gg3o","outputId":"a0f9fdcf-a53d-455d-bd49-691f1f737a47","execution":{"iopub.status.busy":"2023-03-01T06:27:13.990888Z","iopub.execute_input":"2023-03-01T06:27:13.991579Z","iopub.status.idle":"2023-03-01T06:27:13.999196Z","shell.execute_reply.started":"2023-03-01T06:27:13.991531Z","shell.execute_reply":"2023-03-01T06:27:13.997918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"inputs","metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1667915578819,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"meslQK7Le86I","outputId":"d6cdab93-2640-43d0-d5fd-b907c201177d","execution":{"iopub.status.busy":"2023-03-01T06:27:14.000524Z","iopub.execute_input":"2023-03-01T06:27:14.000828Z","iopub.status.idle":"2023-03-01T06:27:14.015618Z","shell.execute_reply.started":"2023-03-01T06:27:14.000799Z","shell.execute_reply":"2023-03-01T06:27:14.014336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_df.question_text.values[0]","metadata":{"executionInfo":{"elapsed":652,"status":"ok","timestamp":1667915621850,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ARTJ68oehHVR","outputId":"83f27c51-7c51-4808-9450-489e5599776b","execution":{"iopub.status.busy":"2023-03-01T06:27:14.017064Z","iopub.execute_input":"2023-03-01T06:27:14.017439Z","iopub.status.idle":"2023-03-01T06:27:14.028645Z","shell.execute_reply.started":"2023-03-01T06:27:14.017406Z","shell.execute_reply":"2023-03-01T06:27:14.027336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df","metadata":{"executionInfo":{"elapsed":1081,"status":"ok","timestamp":1667915683093,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"fJ497jJyhCaK","outputId":"cdb77624-843c-4fe1-ff52-5f93794be092","execution":{"iopub.status.busy":"2023-03-01T06:27:14.044771Z","iopub.execute_input":"2023-03-01T06:27:14.045461Z","iopub.status.idle":"2023-03-01T06:27:14.059152Z","shell.execute_reply.started":"2023-03-01T06:27:14.045421Z","shell.execute_reply":"2023-03-01T06:27:14.057818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntest_inputs = vectorizer.transform(test_df.question_text)","metadata":{"executionInfo":{"elapsed":112943,"status":"ok","timestamp":1667915815920,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"UDu0wUf9g--d","outputId":"3db731b6-8002-435a-bfb5-7500c8921da8","execution":{"iopub.status.busy":"2023-03-01T06:27:14.060691Z","iopub.execute_input":"2023-03-01T06:27:14.061098Z","iopub.status.idle":"2023-03-01T06:29:21.180232Z","shell.execute_reply.started":"2023-03-01T06:27:14.061059Z","shell.execute_reply":"2023-03-01T06:29:21.178824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"CKLhsg9GdkDW"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ML Models for Text Classification\n\nOutline:\n\n- Create a training & validation set\n- Train a logistic regression model\n- Make predictions on training, validation & test data","metadata":{"id":"AyVwmOboQoJn"}},{"cell_type":"markdown","source":"### Split into Training and Validation Set","metadata":{"id":"1CPsktK-iPgK"}},{"cell_type":"code","source":"sample_df","metadata":{"executionInfo":{"elapsed":3,"status":"ok","timestamp":1667915931527,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"hoIv721biUVy","outputId":"73a07b8c-ec27-495e-8ccd-9bb4c53e40da","execution":{"iopub.status.busy":"2023-03-01T06:29:21.182114Z","iopub.execute_input":"2023-03-01T06:29:21.182953Z","iopub.status.idle":"2023-03-01T06:29:21.199401Z","shell.execute_reply.started":"2023-03-01T06:29:21.182900Z","shell.execute_reply":"2023-03-01T06:29:21.198373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"inputs.shape","metadata":{"executionInfo":{"elapsed":613,"status":"ok","timestamp":1667915949798,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ckPyKMcMRBET","outputId":"0bb74251-5275-42ed-b60a-eb763113e3c0","execution":{"iopub.status.busy":"2023-03-01T06:29:21.203236Z","iopub.execute_input":"2023-03-01T06:29:21.203678Z","iopub.status.idle":"2023-03-01T06:29:21.213031Z","shell.execute_reply.started":"2023-03-01T06:29:21.203639Z","shell.execute_reply":"2023-03-01T06:29:21.211904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split","metadata":{"executionInfo":{"elapsed":348,"status":"ok","timestamp":1667916033391,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"OB2jqu7dirOn","execution":{"iopub.status.busy":"2023-03-01T06:29:21.214538Z","iopub.execute_input":"2023-03-01T06:29:21.215276Z","iopub.status.idle":"2023-03-01T06:29:21.222561Z","shell.execute_reply.started":"2023-03-01T06:29:21.215242Z","shell.execute_reply":"2023-03-01T06:29:21.221295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_inputs, val_inputs, train_targets, val_targets = train_test_split(inputs, sample_df.target, \n                                                                        test_size=0.3, random_state=42)","metadata":{"executionInfo":{"elapsed":364,"status":"ok","timestamp":1667916089349,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"m3eE_6y6ieLP","execution":{"iopub.status.busy":"2023-03-01T06:29:21.224439Z","iopub.execute_input":"2023-03-01T06:29:21.224853Z","iopub.status.idle":"2023-03-01T06:29:21.559362Z","shell.execute_reply.started":"2023-03-01T06:29:21.224819Z","shell.execute_reply":"2023-03-01T06:29:21.558145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_inputs.shape","metadata":{"executionInfo":{"elapsed":908,"status":"ok","timestamp":1667916115211,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"1hMoWlcrRBBt","outputId":"c604cdcc-ed79-4c04-dba0-8d46713cab38","execution":{"iopub.status.busy":"2023-03-01T06:29:21.561373Z","iopub.execute_input":"2023-03-01T06:29:21.561745Z","iopub.status.idle":"2023-03-01T06:29:21.568185Z","shell.execute_reply.started":"2023-03-01T06:29:21.561710Z","shell.execute_reply":"2023-03-01T06:29:21.567048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_targets.shape","metadata":{"executionInfo":{"elapsed":986,"status":"ok","timestamp":1667916129267,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ioOd41xKjCkX","outputId":"f3c280ca-aa5c-4f49-d551-3ffebe05c89a","execution":{"iopub.status.busy":"2023-03-01T06:29:21.569808Z","iopub.execute_input":"2023-03-01T06:29:21.570253Z","iopub.status.idle":"2023-03-01T06:29:21.579936Z","shell.execute_reply.started":"2023-03-01T06:29:21.570204Z","shell.execute_reply":"2023-03-01T06:29:21.578857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_inputs.shape","metadata":{"executionInfo":{"elapsed":493,"status":"ok","timestamp":1667916138104,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"d2ipJkxRjChw","outputId":"798265ae-8305-41b5-f96c-8611c45aef4f","execution":{"iopub.status.busy":"2023-03-01T06:29:21.581398Z","iopub.execute_input":"2023-03-01T06:29:21.581713Z","iopub.status.idle":"2023-03-01T06:29:21.593506Z","shell.execute_reply.started":"2023-03-01T06:29:21.581684Z","shell.execute_reply":"2023-03-01T06:29:21.592243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_targets.shape","metadata":{"executionInfo":{"elapsed":1040,"status":"ok","timestamp":1667916148487,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"TZuuLryBjIWZ","outputId":"caa99599-0889-4588-ec20-3ad21881b1d6","execution":{"iopub.status.busy":"2023-03-01T06:29:21.595207Z","iopub.execute_input":"2023-03-01T06:29:21.595569Z","iopub.status.idle":"2023-03-01T06:29:21.603970Z","shell.execute_reply.started":"2023-03-01T06:29:21.595531Z","shell.execute_reply":"2023-03-01T06:29:21.603127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train Logistic Regression model","metadata":{"id":"bBAcDSenjMSk"}},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression","metadata":{"executionInfo":{"elapsed":535,"status":"ok","timestamp":1667916194680,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"tURg-O4pjMmX","execution":{"iopub.status.busy":"2023-03-01T06:29:21.604947Z","iopub.execute_input":"2023-03-01T06:29:21.605290Z","iopub.status.idle":"2023-03-01T06:29:21.615589Z","shell.execute_reply.started":"2023-03-01T06:29:21.605260Z","shell.execute_reply":"2023-03-01T06:29:21.614501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MAX_ITER = 1000","metadata":{"executionInfo":{"elapsed":373,"status":"ok","timestamp":1667918920336,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"aqk9RFIYj9tu","execution":{"iopub.status.busy":"2023-03-01T06:29:21.616949Z","iopub.execute_input":"2023-03-01T06:29:21.617676Z","iopub.status.idle":"2023-03-01T06:29:21.625379Z","shell.execute_reply.started":"2023-03-01T06:29:21.617632Z","shell.execute_reply":"2023-03-01T06:29:21.624216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = LogisticRegression(max_iter=MAX_ITER, solver='sag')","metadata":{"executionInfo":{"elapsed":626,"status":"ok","timestamp":1667916401162,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"uUQ0wzNIjNHr","execution":{"iopub.status.busy":"2023-03-01T06:29:21.626700Z","iopub.execute_input":"2023-03-01T06:29:21.627008Z","iopub.status.idle":"2023-03-01T06:29:21.637326Z","shell.execute_reply.started":"2023-03-01T06:29:21.626965Z","shell.execute_reply":"2023-03-01T06:29:21.636211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nmodel.fit(train_inputs, train_targets)","metadata":{"executionInfo":{"elapsed":27440,"status":"ok","timestamp":1667916434501,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"fOehu5rLjNFl","outputId":"319e0d60-e7db-4851-d7a5-1bcefd58f239","execution":{"iopub.status.busy":"2023-03-01T06:29:21.638643Z","iopub.execute_input":"2023-03-01T06:29:21.639116Z","iopub.status.idle":"2023-03-01T06:41:27.294639Z","shell.execute_reply.started":"2023-03-01T06:29:21.639069Z","shell.execute_reply":"2023-03-01T06:41:27.293385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Make predictions using the model","metadata":{"id":"o16-8HmXmQT0"}},{"cell_type":"code","source":"train_preds = model.predict(train_inputs)","metadata":{"executionInfo":{"elapsed":1020,"status":"ok","timestamp":1667916460185,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"i12DBlkVkThU","execution":{"iopub.status.busy":"2023-03-01T06:41:27.296186Z","iopub.execute_input":"2023-03-01T06:41:27.296486Z","iopub.status.idle":"2023-03-01T06:41:27.328300Z","shell.execute_reply.started":"2023-03-01T06:41:27.296458Z","shell.execute_reply":"2023-03-01T06:41:27.327344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_targets","metadata":{"executionInfo":{"elapsed":363,"status":"ok","timestamp":1667916573761,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"urNCyppPkxZB","outputId":"71349a02-b3a6-4b44-d87f-727dde95605e","execution":{"iopub.status.busy":"2023-03-01T06:41:27.329749Z","iopub.execute_input":"2023-03-01T06:41:27.330526Z","iopub.status.idle":"2023-03-01T06:41:27.338477Z","shell.execute_reply.started":"2023-03-01T06:41:27.330453Z","shell.execute_reply":"2023-03-01T06:41:27.337366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_preds","metadata":{"executionInfo":{"elapsed":861,"status":"ok","timestamp":1667916464475,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"qS8PDYxljNDP","outputId":"9216c5a9-c9aa-4d6a-d540-e804d4d3c925","execution":{"iopub.status.busy":"2023-03-01T06:41:27.339935Z","iopub.execute_input":"2023-03-01T06:41:27.340402Z","iopub.status.idle":"2023-03-01T06:41:27.351443Z","shell.execute_reply.started":"2023-03-01T06:41:27.340359Z","shell.execute_reply":"2023-03-01T06:41:27.350498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series(train_preds).value_counts()","metadata":{"executionInfo":{"elapsed":567,"status":"ok","timestamp":1667916484628,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"W74eTiJgjNA9","outputId":"4f1aa29b-d959-4561-9384-a83fc36ca13d","execution":{"iopub.status.busy":"2023-03-01T06:41:27.352921Z","iopub.execute_input":"2023-03-01T06:41:27.353226Z","iopub.status.idle":"2023-03-01T06:41:27.372774Z","shell.execute_reply.started":"2023-03-01T06:41:27.353198Z","shell.execute_reply":"2023-03-01T06:41:27.371594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series(train_targets).value_counts()","metadata":{"executionInfo":{"elapsed":1125,"status":"ok","timestamp":1667916512008,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"pixsvLxrkhK4","outputId":"44819bce-587f-41e3-cb6b-55e67b8c7c84","execution":{"iopub.status.busy":"2023-03-01T06:41:27.374206Z","iopub.execute_input":"2023-03-01T06:41:27.374538Z","iopub.status.idle":"2023-03-01T06:41:27.395189Z","shell.execute_reply.started":"2023-03-01T06:41:27.374500Z","shell.execute_reply":"2023-03-01T06:41:27.394300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import accuracy_score","metadata":{"executionInfo":{"elapsed":370,"status":"ok","timestamp":1667916547769,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"YxH33IpFkpXJ","execution":{"iopub.status.busy":"2023-03-01T06:41:27.397086Z","iopub.execute_input":"2023-03-01T06:41:27.397570Z","iopub.status.idle":"2023-03-01T06:41:27.403354Z","shell.execute_reply.started":"2023-03-01T06:41:27.397526Z","shell.execute_reply":"2023-03-01T06:41:27.402256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_score(train_targets, train_preds)","metadata":{"executionInfo":{"elapsed":893,"status":"ok","timestamp":1667916561938,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"oUgKam9RkYD6","outputId":"dfc12873-4db8-4993-ad67-bbf6456d8b18","execution":{"iopub.status.busy":"2023-03-01T06:41:27.405234Z","iopub.execute_input":"2023-03-01T06:41:27.405737Z","iopub.status.idle":"2023-03-01T06:41:27.483539Z","shell.execute_reply.started":"2023-03-01T06:41:27.405692Z","shell.execute_reply":"2023-03-01T06:41:27.482361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np","metadata":{"executionInfo":{"elapsed":373,"status":"ok","timestamp":1667916643295,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ipzg1inClCcJ","execution":{"iopub.status.busy":"2023-03-01T06:41:27.485309Z","iopub.execute_input":"2023-03-01T06:41:27.485744Z","iopub.status.idle":"2023-03-01T06:41:27.491814Z","shell.execute_reply.started":"2023-03-01T06:41:27.485700Z","shell.execute_reply":"2023-03-01T06:41:27.489927Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_score(train_targets, np.zeros(len(train_targets)))","metadata":{"executionInfo":{"elapsed":620,"status":"ok","timestamp":1667916647313,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"_P-IsOtTk5x_","outputId":"3aed16ea-c1ab-4c6c-db22-cc083ebe1278","execution":{"iopub.status.busy":"2023-03-01T06:41:27.493352Z","iopub.execute_input":"2023-03-01T06:41:27.493665Z","iopub.status.idle":"2023-03-01T06:41:27.609705Z","shell.execute_reply.started":"2023-03-01T06:41:27.493636Z","shell.execute_reply":"2023-03-01T06:41:27.608571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import f1_score","metadata":{"executionInfo":{"elapsed":896,"status":"ok","timestamp":1667916682226,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"89TtZhqclKwb","execution":{"iopub.status.busy":"2023-03-01T06:41:27.611127Z","iopub.execute_input":"2023-03-01T06:41:27.611417Z","iopub.status.idle":"2023-03-01T06:41:27.615753Z","shell.execute_reply.started":"2023-03-01T06:41:27.611389Z","shell.execute_reply":"2023-03-01T06:41:27.615017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f1_score(train_targets, train_preds)","metadata":{"executionInfo":{"elapsed":476,"status":"ok","timestamp":1667916695952,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"pim_sjENlKn0","outputId":"a919cf7d-3c6b-42ce-c99e-e2da856291ab","execution":{"iopub.status.busy":"2023-03-01T06:41:27.617428Z","iopub.execute_input":"2023-03-01T06:41:27.618165Z","iopub.status.idle":"2023-03-01T06:41:27.918897Z","shell.execute_reply.started":"2023-03-01T06:41:27.618088Z","shell.execute_reply":"2023-03-01T06:41:27.917781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f1_score(train_targets, np.zeros(len(train_targets)))","metadata":{"executionInfo":{"elapsed":375,"status":"ok","timestamp":1667916720434,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"RembBU-Fksbk","outputId":"9c815608-6f06-45d9-c352-d52ad659c965","execution":{"iopub.status.busy":"2023-03-01T06:41:27.922569Z","iopub.execute_input":"2023-03-01T06:41:27.923023Z","iopub.status.idle":"2023-03-01T06:41:28.363458Z","shell.execute_reply.started":"2023-03-01T06:41:27.922988Z","shell.execute_reply":"2023-03-01T06:41:28.362267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_preds = np.random.choice((0, 1), len(train_targets))\nf1_score(train_targets, random_preds)","metadata":{"executionInfo":{"elapsed":530,"status":"ok","timestamp":1667916804410,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"43hI8WB8ld0e","outputId":"36a47c48-a715-47a9-9698-92c5eec0cd7f","execution":{"iopub.status.busy":"2023-03-01T06:41:28.364747Z","iopub.execute_input":"2023-03-01T06:41:28.365117Z","iopub.status.idle":"2023-03-01T06:41:28.735997Z","shell.execute_reply.started":"2023-03-01T06:41:28.365066Z","shell.execute_reply":"2023-03-01T06:41:28.734807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_preds = model.predict(val_inputs)","metadata":{"executionInfo":{"elapsed":394,"status":"ok","timestamp":1667916889103,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"FM0X5ImAl7gB","execution":{"iopub.status.busy":"2023-03-01T06:41:28.737404Z","iopub.execute_input":"2023-03-01T06:41:28.737757Z","iopub.status.idle":"2023-03-01T06:41:28.753890Z","shell.execute_reply.started":"2023-03-01T06:41:28.737726Z","shell.execute_reply":"2023-03-01T06:41:28.752851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_score(val_targets, val_preds)","metadata":{"executionInfo":{"elapsed":592,"status":"ok","timestamp":1667916920449,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"WiXz18lal7db","outputId":"1199b1c7-51b2-4c95-d83e-d945e1541bf9","execution":{"iopub.status.busy":"2023-03-01T06:41:28.755201Z","iopub.execute_input":"2023-03-01T06:41:28.755598Z","iopub.status.idle":"2023-03-01T06:41:28.789033Z","shell.execute_reply.started":"2023-03-01T06:41:28.755562Z","shell.execute_reply":"2023-03-01T06:41:28.788194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f1_score(val_targets, val_preds)","metadata":{"executionInfo":{"elapsed":902,"status":"ok","timestamp":1667916932147,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"Kj-oIhIHksY-","outputId":"de4648a5-5f9c-48fb-f86e-aad601d0a956","execution":{"iopub.status.busy":"2023-03-01T06:41:28.790223Z","iopub.execute_input":"2023-03-01T06:41:28.790747Z","iopub.status.idle":"2023-03-01T06:41:28.917333Z","shell.execute_reply.started":"2023-03-01T06:41:28.790714Z","shell.execute_reply":"2023-03-01T06:41:28.916266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sincere_df.question_text.values[:10]","metadata":{"executionInfo":{"elapsed":653,"status":"ok","timestamp":1667917017459,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"qxCSeXZQmHpS","outputId":"7cb895cb-7605-45d2-b650-b61b02cecffe","execution":{"iopub.status.busy":"2023-03-01T06:41:28.918533Z","iopub.execute_input":"2023-03-01T06:41:28.918837Z","iopub.status.idle":"2023-03-01T06:41:28.925559Z","shell.execute_reply.started":"2023-03-01T06:41:28.918808Z","shell.execute_reply":"2023-03-01T06:41:28.924336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sincere_df.target.values[:10]","metadata":{"executionInfo":{"elapsed":1051,"status":"ok","timestamp":1667917031436,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"fwlbPtLDmgDf","outputId":"a6360495-7f8f-489e-b955-56fbcffa88bd","execution":{"iopub.status.busy":"2023-03-01T06:41:28.927496Z","iopub.execute_input":"2023-03-01T06:41:28.928233Z","iopub.status.idle":"2023-03-01T06:41:28.937458Z","shell.execute_reply.started":"2023-03-01T06:41:28.928186Z","shell.execute_reply":"2023-03-01T06:41:28.936241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.predict(vectorizer.transform(sincere_df.question_text.values[:10]))","metadata":{"executionInfo":{"elapsed":375,"status":"ok","timestamp":1667917062825,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"qHE1gR0mmV0l","outputId":"c1e5f7f9-e5a3-4ea7-ff65-caa1c9280ce0","execution":{"iopub.status.busy":"2023-03-01T06:41:28.939336Z","iopub.execute_input":"2023-03-01T06:41:28.939734Z","iopub.status.idle":"2023-03-01T06:41:28.952173Z","shell.execute_reply.started":"2023-03-01T06:41:28.939694Z","shell.execute_reply":"2023-03-01T06:41:28.951373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"insincere_df.question_text.values[:10]","metadata":{"executionInfo":{"elapsed":411,"status":"ok","timestamp":1667917165691,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"k58qFC-kmVyR","outputId":"d6b34cf6-182e-4375-d37e-01c3cfae8844","execution":{"iopub.status.busy":"2023-03-01T06:41:28.953319Z","iopub.execute_input":"2023-03-01T06:41:28.954308Z","iopub.status.idle":"2023-03-01T06:41:28.965132Z","shell.execute_reply.started":"2023-03-01T06:41:28.954274Z","shell.execute_reply":"2023-03-01T06:41:28.964169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"insincere_df.target.values[:10]","metadata":{"executionInfo":{"elapsed":389,"status":"ok","timestamp":1667917180899,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"lFwis7sOmVv7","outputId":"f5b1eb04-8277-4b61-e0d8-6a08b7840770","execution":{"iopub.status.busy":"2023-03-01T06:41:28.966682Z","iopub.execute_input":"2023-03-01T06:41:28.967319Z","iopub.status.idle":"2023-03-01T06:41:28.977556Z","shell.execute_reply.started":"2023-03-01T06:41:28.967272Z","shell.execute_reply":"2023-03-01T06:41:28.976227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.predict(vectorizer.transform(insincere_df.question_text.values[:10]))","metadata":{"executionInfo":{"elapsed":572,"status":"ok","timestamp":1667917197399,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"2T_zvDYvmVt6","outputId":"36b18600-75b3-4287-de0d-b963b97602f6","execution":{"iopub.status.busy":"2023-03-01T06:41:28.979326Z","iopub.execute_input":"2023-03-01T06:41:28.979697Z","iopub.status.idle":"2023-03-01T06:41:28.995050Z","shell.execute_reply.started":"2023-03-01T06:41:28.979654Z","shell.execute_reply":"2023-03-01T06:41:28.993869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Make Predictions and Submit to Kaggle","metadata":{"id":"TSFA3r48Q2ec"}},{"cell_type":"code","source":"test_df","metadata":{"executionInfo":{"elapsed":1103,"status":"ok","timestamp":1667917318912,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"bIAZhmcWQDd4","outputId":"e92eb5e4-41ad-4590-da74-d9b14dc9697a","execution":{"iopub.status.busy":"2023-03-01T06:41:28.998087Z","iopub.execute_input":"2023-03-01T06:41:28.998462Z","iopub.status.idle":"2023-03-01T06:41:29.013678Z","shell.execute_reply.started":"2023-03-01T06:41:28.998413Z","shell.execute_reply":"2023-03-01T06:41:29.012307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_inputs.shape","metadata":{"executionInfo":{"elapsed":460,"status":"ok","timestamp":1667917332470,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"jzb8x69DnoHz","outputId":"23a5695e-d7b4-49ae-c715-2360925ad007","execution":{"iopub.status.busy":"2023-03-01T06:41:29.015114Z","iopub.execute_input":"2023-03-01T06:41:29.015501Z","iopub.status.idle":"2023-03-01T06:41:29.022960Z","shell.execute_reply.started":"2023-03-01T06:41:29.015468Z","shell.execute_reply":"2023-03-01T06:41:29.021825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_preds = model.predict(test_inputs)","metadata":{"executionInfo":{"elapsed":1112,"status":"ok","timestamp":1667917351071,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"oiY8Ai-anqRP","execution":{"iopub.status.busy":"2023-03-01T06:41:29.024563Z","iopub.execute_input":"2023-03-01T06:41:29.025570Z","iopub.status.idle":"2023-03-01T06:41:29.043573Z","shell.execute_reply.started":"2023-03-01T06:41:29.025513Z","shell.execute_reply":"2023-03-01T06:41:29.042699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df","metadata":{"executionInfo":{"elapsed":558,"status":"ok","timestamp":1667917386054,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"W8h1xhbZnwBt","outputId":"6698c104-0bd3-4c8a-b12f-6e59feac67a3","execution":{"iopub.status.busy":"2023-03-01T06:41:29.045056Z","iopub.execute_input":"2023-03-01T06:41:29.045980Z","iopub.status.idle":"2023-03-01T06:41:29.061676Z","shell.execute_reply.started":"2023-03-01T06:41:29.045928Z","shell.execute_reply":"2023-03-01T06:41:29.060850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df.prediction = test_preds","metadata":{"executionInfo":{"elapsed":4,"status":"ok","timestamp":1667917429135,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"t6xxbg4yn9pT","execution":{"iopub.status.busy":"2023-03-01T06:41:29.063068Z","iopub.execute_input":"2023-03-01T06:41:29.064120Z","iopub.status.idle":"2023-03-01T06:41:29.070861Z","shell.execute_reply.started":"2023-03-01T06:41:29.064074Z","shell.execute_reply":"2023-03-01T06:41:29.069687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df.prediction.value_counts()","metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1667917437994,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"P1vWhwOIn4id","outputId":"4822357f-2f37-4a73-c939-ca1d3f981454","execution":{"iopub.status.busy":"2023-03-01T06:41:29.072342Z","iopub.execute_input":"2023-03-01T06:41:29.073377Z","iopub.status.idle":"2023-03-01T06:41:29.088567Z","shell.execute_reply.started":"2023-03-01T06:41:29.073329Z","shell.execute_reply":"2023-03-01T06:41:29.087259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df","metadata":{"executionInfo":{"elapsed":371,"status":"ok","timestamp":1667917442478,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"ZSHxruEXoFWj","outputId":"231bcd4a-0219-46cc-935a-a83b4e501672","execution":{"iopub.status.busy":"2023-03-01T06:41:29.090241Z","iopub.execute_input":"2023-03-01T06:41:29.090719Z","iopub.status.idle":"2023-03-01T06:41:29.108024Z","shell.execute_reply.started":"2023-03-01T06:41:29.090683Z","shell.execute_reply":"2023-03-01T06:41:29.106720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df.to_csv('submission.csv', index=None)","metadata":{"executionInfo":{"elapsed":771,"status":"ok","timestamp":1667917485705,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"r_7JT4-JoGkL","execution":{"iopub.status.busy":"2023-03-01T06:41:29.111914Z","iopub.execute_input":"2023-03-01T06:41:29.112501Z","iopub.status.idle":"2023-03-01T06:41:29.637893Z","shell.execute_reply.started":"2023-03-01T06:41:29.112450Z","shell.execute_reply":"2023-03-01T06:41:29.636925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!head submission.csv","metadata":{"executionInfo":{"elapsed":1077,"status":"ok","timestamp":1667917506958,"user":{"displayName":"Aakash N S","userId":"03254185060287524023"},"user_tz":-330},"id":"496wLhWjoRA3","outputId":"24c74944-ee72-4168-8342-0c52decb77c4","execution":{"iopub.status.busy":"2023-03-01T06:41:29.639454Z","iopub.execute_input":"2023-03-01T06:41:29.640181Z","iopub.status.idle":"2023-03-01T06:41:30.782596Z","shell.execute_reply.started":"2023-03-01T06:41:29.640136Z","shell.execute_reply":"2023-03-01T06:41:30.781481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"o84S545HoWE4"},"execution_count":null,"outputs":[]}]}