{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Abstract

\nThis notebook illustrates a model developed to tackle Quora Insincere Questions Classification. I employ supervised learning models based on Logistics Regression. My late submission of the dedicated competition approximately reached the score of 0.54.","metadata":{"papermill":{"duration":0.049279,"end_time":"2021-08-27T08:47:17.846091","exception":false,"start_time":"2021-08-27T08:47:17.796812","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Import needed libraries: Numpy, Pandas, Matplotlib, Seaborn\nimport numpy as np\nimport pandas as pd\n%matplotlib inline\nimport matplotlib as mp\nimport matplotlib.pyplot as plt\nimport seaborn as sns","metadata":{"execution":{"iopub.execute_input":"2021-08-27T08:47:17.943788Z","iopub.status.busy":"2021-08-27T08:47:17.942565Z","iopub.status.idle":"2021-08-27T08:47:18.998137Z","shell.execute_reply":"2021-08-27T08:47:18.99729Z"},"papermill":{"duration":1.105332,"end_time":"2021-08-27T08:47:18.998328","exception":false,"start_time":"2021-08-27T08:47:17.892996","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Introduction

\n

Problem Description

\n \nAn existential problem for any major website today is how to handle toxic and divisive content. Quora wants to tackle this problem head-on to keep their platform a place where users can feel safe sharing their knowledge with the world.\n\nQuora is a platform that empowers people to learn from each other. On Quora, people can ask questions and connect with others who contribute unique insights and quality answers. A key challenge is to weed out insincere questions - those founded upon false premises, or that intend to make a statement rather than look for helpful answers.\n\nIn this competition, Kagglers will develop models that identify and flag insincere questions. To date, Quora has employed both machine learning and manual review to address this problem. More scalable methods could be developed to detect toxic and misleading content.\n","metadata":{"papermill":{"duration":0.045653,"end_time":"2021-08-27T08:47:19.089848","exception":false,"start_time":"2021-08-27T08:47:19.044195","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"

Data Description

\nIn this competition the model should be able to detect whether a question asked on Quora is sincere or not. An insincere question is defined as a question intended to make a statement rather than look for helpful answers. Some characteristics that can signify that a question is insincere:\n\n* Has a non-neutral tone\n * Has an exaggerated tone to underscore a point about a group of people\n * Is rhetorical and meant to imply a statement about a group of people\n* Is disparaging or inflammatory\n * Suggests a discriminatory idea against a protected class of people, or seeks confirmation of a stereotype\n * Makes disparaging attacks/insults against a specific person or group of people\n * Based on an outlandish premise about a group of people\n * Disparages against a characteristic that is not fixable and not measurable\n* Isn't grounded in reality\n * Based on false information, or contains absurd assumptions\n* Uses sexual content (incest, bestiality, pedophilia) for shock value, and not to seek genuine answers\n\nThe training data includes the question that was asked, and whether it was identified as insincere (target = 1) or not (target = 0).","metadata":{"papermill":{"duration":0.044791,"end_time":"2021-08-27T08:47:19.179978","exception":false,"start_time":"2021-08-27T08:47:19.135187","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Read data and show the first 5 rows:\ndata_raw = pd.read_csv('/kaggle/input/quora-insincere-questions-classification/train.csv')\ndata_raw.head()","metadata":{"execution":{"iopub.execute_input":"2021-08-27T08:47:19.275039Z","iopub.status.busy":"2021-08-27T08:47:19.274353Z","iopub.status.idle":"2021-08-27T08:47:24.03998Z","shell.execute_reply":"2021-08-27T08:47:24.039369Z"},"papermill":{"duration":4.815494,"end_time":"2021-08-27T08:47:24.040184","exception":false,"start_time":"2021-08-27T08:47:19.22469","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's take a look at the questions in this dataset and try finding out how they have been classified as sincere and insincere.","metadata":{"papermill":{"duration":0.049238,"end_time":"2021-08-27T08:47:24.138543","exception":false,"start_time":"2021-08-27T08:47:24.089305","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Insincere questions example\ninsincere_questions.sample(3, random_state=1).values","metadata":{"execution":{"iopub.execute_input":"2021-08-27T08:47:24.241704Z","iopub.status.busy":"2021-08-27T08:47:24.240827Z","iopub.status.idle":"2021-08-27T08:47:24.322788Z","shell.execute_reply":"2021-08-27T08:47:24.321416Z"},"papermill":{"duration":0.135226,"end_time":"2021-08-27T08:47:24.32316","exception":true,"start_time":"2021-08-27T08:47:24.187934","status":"failed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Sincere questions example\nsincere_questions.sample(3, random_state=1).values","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Related Works

\n\n* Improve your Score with some Text Preprocessing (v1 and v2) by @theoviel: \nhttps://www.kaggle.com/theoviel/improve-your-score-with-some-text-preprocessing |\nhttps://www.kaggle.com/theoviel/improve-your-score-with-text-preprocessing-v2\n* More Text Cleaning To Increase Word Coverage by @sunnymarkliu:\nhttps://www.kaggle.com/sunnymarkliu/more-text-cleaning-to-increase-word-coverage\n* Baseline Model: Logistics Regression by @saket7788:\nhttps://www.kaggle.com/saket7788/baseline-model-logistic-regression\n\nI have also researched on multiple approaches such as GRU, LSTM, etc. but have not applied these to tackle the challenge. ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"

Data Analysis and Visualization

","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"

Raw Data Analysis

","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"It can be easily detected that the dataset is imbalanced, in which the vast majority of questions are sincere, and only a small number are insincere.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"# Count the values of sincere and insincere questions labeled\nvalues = data_raw.target.value_counts()\nprint(values)\n\n# Calculate the percentage of sincere and insincere questions labeled\nsincere_q_pc = values[0]/values.sum()*100\ninsincere_q_pc = values[1]/values.sum()*100\nprint('\\n{}% of questions are sincere while {}% are insincere'.format(sincere_q_pc, insincere_q_pc))","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Draw a graph to illustrate sincere and insincere questions\nnames = ['Sincere', 'Insincere']\n\nplt.bar(names, values)\nplt.suptitle('Number of Sincere and Insincere Questions')\nplt.show()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To find the most frequently occuring words in questions, I built word clouds of a random sample of 1000 insincere and 1000 sincere questions.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"insincere_questions = data_raw[data_raw['target'] == 1].question_text\nsincere_questions = data_raw[data_raw['target'] == 0].question_text","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Import the wordcloud library\nfrom wordcloud import WordCloud, ImageColorGenerator\n\n# Split sentences into a dictionary of uniquely occuring words and their frequencies\ndef word_freq_dict(text):\n # Convert text into word list\n wordList = text.split()\n # Generate word freq dictionary\n wordFreqDict = {word: wordList.count(word) for word in wordList}\n return wordFreqDict","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot a wordcloud from a word frequency dictionary\ndef word_cloud_from_frequency(word_freq_dict, title, figure_size=(10,6)):\n wordcloud.generate_from_frequencies(word_freq_dict)\n plt.figure(figsize=figure_size)\n plt.imshow(wordcloud)\n plt.axis(\"off\")\n plt.title(title)\n plt.show()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Wordcloud of a random sample of 1000 insincere questions\ninsincere_questions = data_raw.question_text[data_raw['target'] == 1]\ninsincere_sample = \" \".join(insincere_questions.sample(1000, random_state=1).values)\ninsincere_word_freq = word_freq_dict(insincere_sample)\nwordcloud = WordCloud(width= 5000,\n height=3000,\n max_words=200,\n colormap='Reds',\n background_color='white')\n\nword_cloud_from_frequency(insincere_word_freq, \"Most Frequent Words in a sample of 1000 raw questions flagged insincere\") ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Wordcloud of a random sample of 1000 sincere questions\nsincere_questions = data_raw.question_text[data_raw['target'] == 0]\nsincere_sample = \" \".join(sincere_questions.sample(1000, random_state=1).values)\nsincere_word_freq = word_freq_dict(sincere_sample)\nwordcloud = WordCloud(width= 5000,\n height=3000,\n max_words=200,\n colormap='Greens',\n background_color='white')\n\nword_cloud_from_frequency(sincere_word_freq, \"Most Frequent Words in a sample of 1000 raw questions flagged sincere\") ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Obviously, predominance words appear in the word cloud are commonly used words such as 'what', 'is', 'with', 'are', etc. which are useless for the model; consequently, these common words (stopwords) needed to be filtered out.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"## **Data Preprocessing**","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"Preprocessing is one of the key steps in every natural language processing problem as it transforms data into usable one which machine can easily interprete. \n\nAs mentioned above, this component have to take out all the stopwords. Moreover, since the input data are raw text from websites, the input text can have noise which can harmful to machine learning performance such as special characters, spelling mistakes, spacing errors, etc. ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"import nltk\nimport sys\nimport spacy\n\n#nltk.download('stopwords')\n#nltk.download('averaged_perceptron_tagger')\n#nltk.download('wordnet')\n\nfrom nltk.corpus import stopwords\nfrom nltk.stem.porter import PorterStemmer\nimport string","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To standardize the data and reduce the amount of noise, preprocessing process is applied according to the following steps:\n\n• Step 1: Lowercase all characters.\n\n• Step 2: Split text into list.\n\n• Step 3: Punctuations were completely removed.\n\n• Step 4: Remove all stop words.\n\n• Step 5: Stem: technique used to extract the base form of the words by removing affixes from them. \n\nNatural Language Toolkit (NLTK) is used for all pre-processing steps thanks to its popularity and simplicity.\n\nTransfer acronyms to the full form of them and translated from context.\nTrim unnecessary spaces made by spacing errors.\nSpecial characters (including punctuations) were completely removed.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"nlp = spacy.load(\"en_core_web_sm\", disable=['parser','ner'])\nstop = set(stopwords.words('english'))\npunc = set(string.punctuation)\n\ndef clean_text(text):\n # Convert the text into lowercase\n text = text.lower()\n # Split into list\n wordList = text.split()\n # Remove punctuation\n wordList = [\"\".join(x for x in word if (x==\"'\")|(x not in punc)) for word in wordList]\n # Remove stop words\n wordList = [word for word in wordList if word not in stop]\n # Stem\n porter = PorterStemmer()\n wordList = [porter.stem(word) for word in wordList]\n\n reformed_sentence = \" \".join(wordList)\n doc = nlp(reformed_sentence)\n return \" \".join([token.lemma_ for token in doc])","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see whether the proposed preprocesing methods work or not:","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"question = data_raw.question_text.sample(1, random_state=1).values[0]\nquestion","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clean_text(question)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we will clean every row of text data by running this function:","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"data_raw['clean_text'] = data_raw['question_text'].astype('str').apply(clean_text)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Build wordcloud to visualize questions after cleaning process by similar block of code with just slightly adjustment: ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"# Wordcloud of a random sample of 1000 cleaned insincere questions\nclean_insincere_questions = data_raw.clean_text[data_raw['target'] == 1]\nclean_insincere_sample = \" \".join(clean_insincere_questions.sample(1000, random_state=1).values)\nclean_insincere_word_freq = word_freq_dict(clean_insincere_sample)\nwordcloud = WordCloud(width= 5000,\n height=3000,\n max_words=200,\n colormap='Reds',\n background_color='white')\n\nword_cloud_from_frequency(insincere_word_freq, \"Most Frequent Words in a sample of 1000 cleaned questions flagged insincere\") ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Wordcloud of a random sample of 1000 clean sincere questions\nclean_sincere_questions = data_raw.clean_text[data_raw['target'] == 0]\nclean_sincere_sample = \" \".join(clean_sincere_questions.sample(1000, random_state=1).values)\nclean_sincere_word_freq = word_freq_dict(clean_sincere_sample)\nwordcloud = WordCloud(width= 5000,\n height=3000,\n max_words=200,\n colormap='Greens',\n background_color='white')\n\nword_cloud_from_frequency(sincere_word_freq, \"Most Frequent Words in a sample of 1000 cleaned questions flagged sincere\") ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

TextToVec

","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"

Bag of Words

","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"It is basic model used in natural language processing. It is called bag of words because any order of the words in the document is discarded, so it only tells us whether word is present in the document or not.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer\nbow_converter = CountVectorizer()\n","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_question_text = data_raw['clean_text'].sample(1, random_state= 1).values\nsample_question_text","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_count_vectorized_data = bow_converter.fit_transform(sample_question_text)\nsample_count_vectorized_data.toarray()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"count_vectorized_data_feature_names = bow_converter.get_feature_names()\ncount_vectorized_data_feature_names","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

TF-IDF

","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"TF-IDF stands for Term Frequency-Inverse Document Frequency which basically tells importance of the word in the corpus or dataset. TF-IDF contain two concept Term Frequency(TF) and Inverse Document Frequency(IDF).\n\nTerm Frequency is defined as how frequently the word appear in the document or corpus. As each sentence is not the same length so it may be possible a word appears in long sentence occur more time as compared to word appear in sorter sentence. Term Frequency can be defined as:\n\nTF = Number of time word appear/Total words\n\nInverse Document frequency is another concept which is used for finding out importance of the word. It is based on the fact that less frequent words are more informative and important. IDF is represented by formula:\n\nIDF = log10(Number of documents/Number of documents contain word)\n\nTF-IDF is basically a multiplication between TF table and IDF table. It basically reduces values of common word that are used in different document.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"from sklearn.feature_extraction.text import TfidfVectorizer\ntfidf_converter = TfidfVectorizer(ngram_range=(1,1))","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_tfidf_vectorized_data = tfidf_converter.fit_transform(sample_question_text)\nsample_tfidf_vectorized_data.toarray()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tfidf_word_feature_names = tfidf_converter.get_feature_names()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tfidf_word_feature_names","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(tfidf_word_feature_names)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"LR is one of the easiest ML algorithms as it is easy to implement, interpret, and very efficient to train. Traning a model with LR doesn’t need high computation effort. LR also less prone to overfitting in a low dimensional dataset, and in context of a higher dimensional dataset, regularization can be used to avoid overfitting. Moreover, new data can be updated using stochastic gradient descent (SGD).\n\nBut LR also has limitations as it only address linear separable data for non-linear problems transformation is required. Features used for training model should also be carefully extracted otherwise noise will make the probabilistic predictions may be incorrect. LR requires a large dataset and sufficient training examples for all the categories it needs to identify. Lastly, each training tuples must be isolated to all others, because relationship between any of them will make model give more importance to these relative examples.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.metrics import accuracy_score, confusion_matrix\nfrom sklearn.model_selection import train_test_split\n\ncount_vectorizer = CountVectorizer()\nmodel = LogisticRegression(C=1, random_state=0)\n\nvectorize_logit_pipeline = Pipeline([\n ('count_vectorizer', count_vectorizer),\n ('logit', model)\n])","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# input variable\nX = data_raw['clean_text']\n# target variable\ny = data_raw['target']","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_X, test_X, train_y, test_y = train_test_split(X, y, test_size=0.3)\nvectorize_logit_pipeline.fit(train_X, train_y)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions = vectorize_logit_pipeline.predict(test_X)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_score(test_y, predictions)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import f1_score\nf1_score(test_y, predictions)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"confusion_matrix_logit_cv = confusion_matrix(test_y, predictions)\nsns.heatmap(confusion_matrix_logit_cv, annot= True, xticklabels=['sincere', 'insincere'], yticklabels=['sincere', 'insincere'])","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import classification_report\nprint(classification_report(test_y, predictions))","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tfidf_ngrams_converter = TfidfVectorizer(ngram_range=(1,2))\ntfidf_ngrams_logit_pipeline = Pipeline([\n ('tfidf_vectorizer', tfidf_ngrams_converter),\n ('logit', model)\n])","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tfidf_ngrams_logit_pipeline.fit(train_X, train_y)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_predictions = tfidf_ngrams_logit_pipeline.predict(test_X)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_score(test_y, new_predictions)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f1_score(test_y, new_predictions)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"confusion_matrix_logit_tfidf = confusion_matrix(test_y, new_predictions)\nsns.heatmap(confusion_matrix_logit_tfidf, annot= True, xticklabels=['sincere', 'insincere'], yticklabels=['sincere', 'insincere'])","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(classification_report(test_y, new_predictions, target_names=['sincere', 'insincere']))","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data = pd.read_csv('/kaggle/input/quora-insincere-questions-classification/test.csv')\ntest_data.head()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data.info()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data['clean_text'] = test_data['question_text'].astype('str').apply(clean_text)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data.head()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_final = test_data['clean_text']","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_final = tfidf_ngrams_logit_pipeline.predict(X_final)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_final[:5]","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data['target'] = y_final","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result_df = test_data[['qid', 'target']]","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result_df.rename(columns={'target': 'prediction'}, inplace=True)\nresult_df.set_index('qid', inplace=True)\nresult_df.head()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result_df.to_csv('submission.csv')\n!head submission.csv","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]}]}