{"cells":[{"metadata":{"_uuid":"c7af27cd9eb94d03c130c1358b73a23db4abbfa9"},"cell_type":"markdown","source":"# Quora Insincere Questions\n\n## Detect toxic content to improve online conversations\n\nAn existential problem for any major website today is how to handle toxic and divisive content. Quora wants to tackle this problem head-on to keep their platform a place where users can feel safe sharing their knowledge with the world.\n\nQuora is a platform that empowers people to learn from each other. On Quora, people can ask questions and connect with others who contribute unique insights and quality answers. A key challenge is to weed out insincere questions -- those founded upon false premises, or that intend to make a statement rather than look for helpful answers.\n\n![quora](https://qph.fs.quoracdn.net/main-qimg-6ba8f7e24e68df3e8d44ed9cf3263fd8)\n\n\n### Lets start having some imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nprint(os.listdir(\"../input\"))\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")\nprint(\"Train shape : \",train_df.shape)\nprint(\"Test shape : \",test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c84e382704612048bdaf86065f1a607df3091c98"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2c0d48dd515c9a771184d49e7941c85065f7c43a"},"cell_type":"code","source":"train_df.target.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"efef52bba9d6a6b46a5123c12229d2e7ae0c8db2"},"cell_type":"code","source":"train_df.question_text.sample(10).values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1f76b2ed69b6990738c884b3b5e2d564d898c17d"},"cell_type":"code","source":"train_df['length']=train_df.question_text.apply(lambda x: len(x))\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"328ff8aa15a51ff60314403e5ac40a515e235ddd"},"cell_type":"code","source":"test_df['length']=test_df.question_text.apply(lambda x: len(x))\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f1d6435466bcb1300ac0a5a7fa7c7a4f31566a2f"},"cell_type":"code","source":"print('Average questions length in train is {0:.0f}.'.format(train_df.length.mean()))\nprint('Average questions length in test is {0:.0f}.'.format(test_df.length.mean()))\nprint()\nprint('Maximum questions length in train is {0:.0f}.'.format(train_df.length.max()))\nprint('Maximum questions length in test is {0:.0f}.'.format(test_df.length.max()))\nprint()\nprint('Minimum questions length in train is {0:.0f}.'.format(train_df.length.min()))\nprint('Minimum questions length in test is {0:.0f}.'.format(test_df.length.min()))\nprint()\nprint('Average word length of questions in train is {0:.0f}.'.format(np.mean(train_df['question_text'].apply(lambda x: len(x.split())))))\nprint('Average word length of questions in test is {0:.0f}.'.format(np.mean(test_df['question_text'].apply(lambda x: len(x.split())))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7dc6fe592a2fbcbffecb7d431e731d51b9475f9e"},"cell_type":"code","source":"print('Max word length of questions in train is {0:.0f}.'.format(np.max(train_df['question_text'].apply(lambda x: len(x.split())))))\nprint('Max word length of questions in test is {0:.0f}.'.format(np.max(test_df['question_text'].apply(lambda x: len(x.split())))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f5a7bab319128d70db969510f7a8bd2b0d0a8a48"},"cell_type":"code","source":"print('Average character length of questions in train is {0:.0f}.'.format(np.mean(train_df['question_text'].apply(lambda x: len(x)))))\nprint('Average character length of questions in test is {0:.0f}.'.format(np.mean(test_df['question_text'].apply(lambda x: len(x)))))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"673800601835768b54ea8b7e233958d24f2cf215"},"cell_type":"markdown","source":"## Lets clean the text\n\nAlthough i find it pretty cleaned, but ofcourse the stopwords needs to be removed and sanitized."},{"metadata":{"trusted":true,"_uuid":"008975af2853ef8645672bfc37cf67cf6767b6ac"},"cell_type":"code","source":"import re\nimport nltk\nfrom nltk.corpus import stopwords\ndef clean_text(raw_text):\n    raw_text=raw_text.strip()\n    try:\n        no_encoding=raw_text.decode(\"utf-8-sig\").replace(u\"\\ufffd\", \"?\")\n    except:\n        no_encoding = raw_text\n    letters_only = re.sub(\"[^a-zA-Z]\", \" \",no_encoding) \n    words = letters_only.lower().split()                             \n    stops = set(stopwords.words(\"english\"))                  \n    meaningful_words = [w for w in words if not w in stops] \n    return( \" \".join( meaningful_words )) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f3cdc7d2bd55f5615d8b1a913ff10764254225a2"},"cell_type":"code","source":"train_df['clean_ques']=train_df.question_text.apply(clean_text)\ntrain_df.sample(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f80cdd7f09fbaec4834a055a97ea5b132aad2d5"},"cell_type":"markdown","source":"## Pipeline Model\n\n### Logistic Regression with CountVectorizer"},{"metadata":{"trusted":true,"_uuid":"12f119763dca32f64b845702fcf2e01b54afeaae"},"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer\nfrom sklearn.pipeline import Pipeline,FeatureUnion\nfrom sklearn.metrics import confusion_matrix,classification_report,f1_score\nfrom sklearn.multiclass import OneVsRestClassifier\nfrom sklearn.naive_bayes import MultinomialNB\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.linear_model import LogisticRegression\n# nb = MultinomialNB()\nft=FeatureUnion([('ct', CountVectorizer(analyzer='char',ngram_range=(1,5),max_df=0.9)),('ct2', CountVectorizer(analyzer='word',ngram_range=(1,4),max_df=0.9))])\npipeline = Pipeline([\n    ('bow',ft),  # strings to token integer counts\n    ('clf', LogisticRegression(solver='saga', class_weight='balanced', C=0.45,max_iter=250, verbose=1))\n])\npipeline.get_params().keys()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"37fa724723522a1d2f9356ee1f1df9ba41d9069e"},"cell_type":"code","source":"pipeline.fit(train_df['clean_ques'].values,train_df.target)\nimport gc\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9320d00a13a6d913ed68508f8c7ea7d7661aec6f"},"cell_type":"code","source":"test_df['clean_ques']=test_df.question_text.apply(clean_text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a25a6a39a70d4661b916b1ae61524bf1864d6751"},"cell_type":"code","source":"pr=pipeline.predict(test_df['clean_ques'].values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f20f81a67f3b2b98052bf6e9e08dc930625d1476"},"cell_type":"code","source":"sub=pd.DataFrame({'qid':test_df.qid,'prediction':pr})\nsub.prediction.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3f920e203235657574b127e816bdfdc5e3538da4"},"cell_type":"code","source":"sub.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"652c92e8e816ae4fc9f82147525b04868b82b8ae"},"cell_type":"markdown","source":"### Will try with Embeddings next!!! Stay Tuned"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}