{"cells":[{"metadata":{"_uuid":"f0924bc3eb5bd2b99ce3d7e666d24db348435ff5"},"cell_type":"markdown","source":"# Quora Insincere Questions - SciKitLearn LR on Histographic Features - Not good for score"},{"metadata":{"_uuid":"7531604b711e95e6c97503f558319db6eb7b2d8c"},"cell_type":"markdown","source":"This Python Notebook is experimental, and was motivated by a histographic-features approach used by some users for some image classification problems.\n\nThe Notebook is not suitable for scoring well on the Kaggle challenge on Quora Insincere Questions.\n\nTo the training and test datasets, the Notebook adds features based on word-frequency histograms.\n\nBy applying the scikit-learn logisti regression method to the training dataset, the Notebook creates a prediction model.\n\nBy applying this prediction model to the test dataset, the Notebook computes predictions."},{"metadata":{"_uuid":"f7beee06ea0bf70505534f547cc3abc8137a262e"},"cell_type":"markdown","source":"Import necessary packages."},{"metadata":{"_uuid":"a349b623f046fe4052c95a3a1cd406d9b5e8a89a","trusted":false},"cell_type":"code","source":"import pandas as pd\nfrom itertools import chain\nfrom collections import Counter\nfrom sklearn.linear_model import LogisticRegression","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"23e8b7ae774fa5a3a2560fe7c0907e551b2ac8c2"},"cell_type":"markdown","source":"Make dataframes from csv files of training dataset and test dataaset."},{"metadata":{"_uuid":"1dd0ac99671ea0af2a11773557cc3b886679dafb","trusted":false},"cell_type":"code","source":"DATA_DIR = '../input/'\nOUT_DIR = ''\n\ntrain_df = pd.read_csv(DATA_DIR + 'train.csv')\ntest_df = pd.read_csv(DATA_DIR + 'test.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"93ad2c7f56db0691250c47e129ef642ff7826441"},"cell_type":"markdown","source":"Define a method to print something in between its title  and separator."},{"metadata":{"_uuid":"c1fe3fb0ca2b5252ec6e467f8493ce9b92cb4718","trusted":false},"cell_type":"code","source":"def print_block(sometitle, someblock):\n    '''\n    Print something in between its title  and separator.\n    '''\n    print(sometitle)\n    print(\"\\n\")\n    print(someblock)\n    print(\"\\n\" + \"=\"*80 + \"\\n\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3cfc8f1f5fcf3a524e182b35701d9cc864debc38"},"cell_type":"markdown","source":"Define a method to trim a text, and split it into one-words and two-words."},{"metadata":{"_uuid":"e6c353e99f6390d77372bfa0e3b9d6ff3a42d1b5","trusted":false},"cell_type":"code","source":"def text_to_words(orig_text):\n    '''\n    Trim a text, and split it into one-words and two-words.\n    '''\n    word_vect = orig_text.lower()\n    word_vect = word_vect.replace(\"  \",\" \")\n    remove_char = ['.', '?', '!', ',', ';', '(', ')', '\"', \"\"\"''\"\"\"]\n    for i in range(9):\n        word_vect = word_vect.replace(remove_char[i], \"\")\n        \n    # Make a list of one words.\n    word_vect = word_vect.split()\n    word_count = len(word_vect)\n    \n    # Add two consecutive words.\n    if word_count > 2:\n        for i in range(word_count - 2):\n            word_vect.append(word_vect[i] + \" \" + word_vect[i+1])\n    \n    # Add two alternate words.\n    if word_count > 3:\n        for i in range(word_count - 3):\n            word_vect.append(word_vect[i] + \" \" + word_vect[i+2])\n    \n    return word_vect","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d0d4f245d4c1fe588c770d7308b7741c10d3227a"},"cell_type":"markdown","source":"Define a method to count matching elements in two dataframes."},{"metadata":{"trusted":false,"_uuid":"4d22c27c7a786b3adead933af3e54d3a6103529e"},"cell_type":"code","source":"def count_match(df1, df2):\n    \"\"\"\n    Count matching elements in two dataframes.\n    \"\"\"\n    qty = 0\n    for ele in df1:\n        if ele in df2:\n            qty = qty + 1\n    return qty","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4298b220bba6261466de954507aedc9efe72dc69"},"cell_type":"markdown","source":"Define a method to prepare a dataframe of most common insincere and sincere words, based on a given dataframe of texts and lables."},{"metadata":{"_uuid":"11057db882a697bda85dd6a98513a2d91ef84109","trusted":false},"cell_type":"code","source":"def df_common_words(df):\n    '''\n    Prepare a dataframe of most common insincere and sincere words,\n    based on a given dataframe of texts and lables.\n    '''\n    insincere_words = df[df['target']==1]['question_text'].apply(\n                   lambda x: text_to_words(x)).reset_index(drop=True)\n    insincere_words = Counter(chain.from_iterable(insincere_words[i] for\n                   i in range(len(insincere_words))))\n    insincere_words = pd.DataFrame(Counter.most_common(insincere_words),\n                      columns=[\"word\",\"freq\"])\n    print_block(\"10 most common insincere words\", insincere_words[:10])\n    \n    sincere_words = df[df['target']==0]['question_text'].apply(\n                   lambda x: text_to_words(x)).reset_index(drop=True)\n    sincere_words = Counter(chain.from_iterable(sincere_words[i] for\n                   i in range(len(sincere_words))))\n    sincere_words = pd.DataFrame(Counter.most_common(sincere_words),\n                    columns=[\"word\",\"freq\"])\n    print_block(\"10 most common sincere words\", sincere_words[:10])\n    \n    insincere_words['both'] = insincere_words['word'].isin(sincere_words['word'])\n    sincere_words['both'] = sincere_words['word'].isin(insincere_words['word'])\n    \n    insincere_words = insincere_words[insincere_words.both == False].reset_index(\n                      drop=True).drop(['freq','both'], axis=1)\n    print_block(\"10 most insincere not-most-sincere words\", insincere_words[:10])\n    \n    sincere_words = sincere_words[sincere_words.both == False].reset_index(\n                    drop=True).drop(['freq','both'], axis=1)\n    print_block(\"10 most sincere not-most-insincere words\", sincere_words[:10])\n    \n    insincere_00 = insincere_words[:1]\n    insincere_01 = insincere_words[1:2]\n    insincere_02 = insincere_words[2:4]\n    insincere_03 = insincere_words[4:8]\n    insincere_04 = insincere_words[8:16]\n    insincere_05 = insincere_words[16:32]\n    insincere_06 = insincere_words[32:64]\n    insincere_07 = insincere_words[64:128]\n    insincere_08 = insincere_words[128:256]\n    insincere_09 = insincere_words[256:512]\n    insincere_10 = insincere_words[512:1024]\n    insincere_99 = insincere_words[1024:]\n    \n    sincere_00 = sincere_words[:1]\n    sincere_01 = sincere_words[1:2]\n    sincere_02 = sincere_words[2:4]\n    sincere_03 = sincere_words[4:8]\n    sincere_04 = sincere_words[8:16]\n    sincere_05 = sincere_words[16:32]\n    sincere_06 = sincere_words[32:64]\n    sincere_07 = sincere_words[64:128]\n    sincere_08 = sincere_words[128:256]\n    sincere_09 = sincere_words[256:512]\n    sincere_10 = sincere_words[512:1024]\n    sincere_11 = sincere_words[1024:2048]\n    sincere_99 = sincere_words[2048:]\n    \n    df_common = (insincere_00, insincere_01, insincere_02, insincere_03, insincere_04,\n                 insincere_05, insincere_06, insincere_07, insincere_08, insincere_09,\n                 insincere_10, insincere_99,\n                 sincere_00, sincere_01, sincere_02, sincere_03, sincere_04, sincere_05,\n                 sincere_06, sincere_07, sincere_08, sincere_09, sincere_10, sincere_11,\n                 sincere_99)\n    \n    return df_common","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f84caa73a2c9a704515c9dd5ffd513a68ef17b66"},"cell_type":"markdown","source":"Prepare a dataframe of words and label-indices, based on the given training dataset."},{"metadata":{"_uuid":"48736a4af595054ac324056290b11eceb516748c","scrolled":false,"trusted":false},"cell_type":"code","source":"df_common = df_common_words(train_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"76493b82aa4ecd3625532744bd14dfd51f89fa4c"},"cell_type":"markdown","source":"Define a method to add various measures to a dataset."},{"metadata":{"trusted":false,"_uuid":"c2bc82cf5464ca0866760cfed7af39d1e6a09667"},"cell_type":"code","source":"def add_measure(df, typ):\n    \"\"\"\n    Add various measures to a dataset.\n    \"\"\"\n    df['chars'] = df['question_text'].apply(lambda x: len(x))\n    df['words'] = df['question_text'].apply(lambda x: len(x.split()))\n    \n    for i in range(25):\n        df['c' + str(100+i)] = df['question_text'].apply(lambda x:\n                         count_match(x.split(\" \"), df_common[i]))\n    \n    dfx = df.drop(['question_text'], axis=1)\n    print(\"First 5 elements of \" + typ + \" dataset:\")\n    print(dfx[:5])\n    \n    return dfx","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"aae43e6901fb6f6f2839df793551b36666784770"},"cell_type":"code","source":"train_df = add_measure(train_df, \"training\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8c3b0cd9f827c19423c3a1f14da5899e623c6663"},"cell_type":"code","source":"test_df = add_measure(test_df, \"test\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3e5b27048341eec24a79085121d66b6d52277677"},"cell_type":"markdown","source":"Develope a prediction model by applying the scikit-learn logistic regression method to the processed training dataframe."},{"metadata":{"_uuid":"d090cae18d941723943a3427f882e3842d171b72","trusted":false},"cell_type":"code","source":"columos = [\"chars\",\"words\",\"c100\",\"c101\",\"c102\",\"c103\",\"c104\",\"c105\",\"c106\",\n           \"c107\",\"c108\",\"c109\",\"c110\",\"c111\",\"c112\",\"c113\",\"c114\",\"c115\",\n           \"c116\",\"c117\",\"c118\",\"c119\",\"c120\",\"c121\",\"c122\",\"c123\",\"c124\"]\n\nclf = LogisticRegression(random_state=0, solver='lbfgs', multi_class='multinomial')\n\nmodel = clf.fit(train_df[columos], train_df['target'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"44f11cc0d33321687001f6ad6e8d6f20fa602094"},"cell_type":"markdown","source":"Compute predictions by applying the developed prediction model to the processed test dataframe."},{"metadata":{"trusted":false,"_uuid":"402189352932b7a8ac15c36aab0a718f4e7df693"},"cell_type":"code","source":"test_df['prediction'] = model.predict(test_df[columos])\n\nprint(test_df['prediction'][:20])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3bb838cc7034e4087dd996450fbd28e5cb16d337"},"cell_type":"markdown","source":"Make csv file of the predicted results as needed for the submission."},{"metadata":{"_uuid":"3ef29be89a944d2ff3a273b6c2e9f4abfeb602a1","trusted":false},"cell_type":"code","source":"test_df[['qid','prediction']].to_csv(OUT_DIR + 'submission.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68ca03d3a3b2b977128fc86b53bcaed91e0e66a6","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}