{"cells":[{"metadata":{"_uuid":"f6609f0380a0b50ce13f48ed76978f8be8803b6d"},"cell_type":"markdown","source":"## Analyse models biggest error predictions\n\nIn this workbook I will show how to print and view the biggest miss-predictions of a classifier.\n\nLooking under the hood, you can see if you can recognize some patterns or specific areas, where classifier is failing - which you could then concentrate on improving."},{"metadata":{"_uuid":"935aeb72a63ed044de81ceada18e583aaf3684dc"},"cell_type":"markdown","source":"## Part 2 after halfway of workbook is where the analysis starts. You might wanna skip to there.\n"},{"metadata":{"_uuid":"e183f04ddc5e57a762edb973502593e0241cfd5a"},"cell_type":"markdown","source":"### Part 1. Training classifier and making predictions"},{"metadata":{"trusted":true,"_uuid":"f842853d00fddf9be5cc3fc88e57ecdb46d49ab1"},"cell_type":"code","source":"import random\nimport pandas as pd\nimport numpy as np\nimport gc\nimport os\nfrom sklearn.metrics import f1_score\nimport re\nimport torch\nimport time\nimport torch.nn as nn\nimport torch.optim as optim\nimport torch.nn.functional as F\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Dense, Input, LSTM, Embedding, Dropout, Activation\nfrom keras.layers import Bidirectional, GlobalMaxPool1D\nfrom keras.models import Model\nfrom keras import initializers, regularizers, constraints, optimizers, layers\nfrom sklearn.model_selection import train_test_split\nfrom tqdm import tqdm_notebook, tnrange\nfrom tqdm.auto import tqdm\ntqdm.pandas(desc='Progress')\n\n#pd.set_option('display.max_colwidth', -1)  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4f95e3055492efa801dbfb7d7ac7c9330fd644cb"},"cell_type":"code","source":"EMBEDDINGS_PATH = '../input/embeddings/'\nEMBEDDING_FILE_GLOVE = f'{EMBEDDINGS_PATH}/glove.840B.300d/glove.840B.300d.txt'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2bd4ad510c8fe4422bdc7d38083aca9862538004"},"cell_type":"code","source":"embed_size = 300 # how big is each word vector\nmax_features = 120000 # how many unique words to use (i.e num rows in embedding vector)\nmaxlen = 50 # max number of words in a question to use\nbatch_size = 1024 # how many samples to process at once\nn_epochs = 2 # how many times to iterate over all samples\nSEED = 1006","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f420d9ecc1335355f6ea9a7f66c785a1083a6ff1"},"cell_type":"code","source":"# REPEATABILITY\ndef seed_everything(seed=SEED):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\nseed_everything()\n# kernel https://www.kaggle.com/hengzheng/pytorch-starter","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7c9fb0bdc7549fe1fb9b797e92fe3116c415494a"},"cell_type":"code","source":"os.environ['OMP_NUM_THREADS'] = '4'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7215339dd0a124ef8c26991a6f9a2a7487fcda93"},"cell_type":"markdown","source":"## Load data"},{"metadata":{"trusted":true,"_uuid":"a679ee4347b88cd901c1e30e1becb0eba2239e39"},"cell_type":"code","source":"df_train = pd.read_csv(\"../input/train.csv\")\ndf_test = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c73b5e358a5e441c83b2aafaac71b706ce65eaa6"},"cell_type":"code","source":"# Randomize\nnp.random.seed(SEED)\ntrn_idx = np.random.permutation(len(df_train))\ndf_train = df_train.iloc[trn_idx]\n\ndf = pd.concat([df_train ,df_test],sort=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"42ad2e4e215ba40d226ab284ad707be007f0ebbb"},"cell_type":"code","source":"df_train.head(3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"251467fd27e6dcad0bcf0aa874f4fd7e62ac7ac9"},"cell_type":"markdown","source":"#### preprocess"},{"metadata":{"trusted":true,"_uuid":"ef5f8edf4cdf3da860357b43078c25a21d2dc9ad"},"cell_type":"code","source":"puncts = [',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', \"'\", '$', '&', '/', '[', ']', '>', '%', '=', '#', '*', '+', '\\\\', '•',  '~', '@', '£', \n '·', '_', '{', '}', '©', '^', '®', '`',  '<', '→', '°', '€', '™', '›',  '♥', '←', '×', '§', '″', '′', 'Â', '█', '½', 'à', '…', \n '“', '★', '”', '–', '●', 'â', '►', '−', '¢', '²', '¬', '░', '¶', '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', \n '▒', '：', '¼', '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲', 'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', \n '∙', '）', '↓', '、', '│', '（', '»', '，', '♪', '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø', '¹', '≤', '‡', '√', ]\n\ndef clean_text(x):\n    x = str(x)\n    for punct in puncts:\n        x = x.replace(punct, f' {punct} ')\n    return x\n\ndf_train[\"question_text\"] = df_train[\"question_text\"].progress_apply(lambda x: clean_text(x))\ndf_test[\"question_text\"] = df_test[\"question_text\"].apply(lambda x: clean_text(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"acac983cf7be4265f2b09dda42ebbe6db0b6dd28"},"cell_type":"code","source":"def clean_numbers(x):\n    x = re.sub('[0-9]{5,}', '#####', x)\n    x = re.sub('[0-9]{4}', '####', x)\n    x = re.sub('[0-9]{3}', '###', x)\n    x = re.sub('[0-9]{2}', '##', x)\n    return x\n\ndf_train[\"question_text\"] = df_train[\"question_text\"].progress_apply(lambda x: clean_numbers(x))\ndf_test[\"question_text\"] = df_test[\"question_text\"].apply(lambda x: clean_numbers(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1e253e75e9f86a5281dffb35218c20ffe22ff090"},"cell_type":"code","source":"specials = {'\\u200b': ' ', '…': ' ... ', '\\ufeff': '', 'करना': '', 'है': ''}\n\ndef clean_special_chars(text):\n    \n    for s in specials:\n        text = text.replace(s, specials[s])    \n    return text\n\ndf_train[\"question_text\"] = df_train[\"question_text\"].progress_apply(lambda x: clean_special_chars(x))\ndf_test[\"question_text\"] = df_test[\"question_text\"].apply(lambda x: clean_special_chars(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3f05037d854d99434d68ef2bf0eac672efdf2c7e"},"cell_type":"code","source":"df_train[\"question_text\"] = df_train[\"question_text\"].apply(lambda x: x.lower())\ndf_test[\"question_text\"] = df_test[\"question_text\"].apply(lambda x: x.lower())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e39cdaf2d82cd42479d9e6f07668f8c8753cbc2d"},"cell_type":"markdown","source":"#### Tokenize"},{"metadata":{"trusted":true,"_uuid":"ff1c33f9bf3f4ecc8ad3f92404ea438f20d4e331"},"cell_type":"code","source":"list_sentences_train = df_train['question_text']\nlist_sentences_test = df_test['question_text']\nlist_sentences_combined = list_sentences_train.append(list_sentences_test, ignore_index=True)\ntokenizer = Tokenizer(num_words=max_features, filters='\\t\\n')\ntokenizer.fit_on_texts(list(list_sentences_combined))\nlist_tokenized_train = tokenizer.texts_to_sequences(list_sentences_train)\nlist_tokenized_test = tokenizer.texts_to_sequences(list_sentences_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f7036f233983ecb4a5d20c546e9349da447e8907"},"cell_type":"code","source":"train_x = pad_sequences(list_tokenized_train, maxlen=maxlen)\ntest_x = pad_sequences(list_tokenized_test, maxlen=maxlen)\ntrain_y = df_train['target'].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f525f84b7263e26516f21ff98c85dbdaa4df8de7"},"cell_type":"markdown","source":"## Embedding"},{"metadata":{"trusted":true,"_uuid":"e3a60e1e38e2c8d41f2079d04773a6bbf044cd10"},"cell_type":"code","source":"start = time.time()\ndef get_coefs(word, *arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.rstrip().rsplit(' ')) for o in open(EMBEDDING_FILE_GLOVE))\nend = time.time()\nprint(end-start)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c9df187abacfaf897f694ff91f1b8c098e3e2dc9"},"cell_type":"code","source":"# Get embedding mean and st deviation for giving random value near mean for words that were not in glove\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nemb_mean,emb_std","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fa9cd4625ae012540d8003065a051bd466a579eb"},"cell_type":"code","source":"word_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0e89bce488ce43746747d0a84e29d8b84dc9070b"},"cell_type":"markdown","source":"#### Simple bidirectional LSTM with two fully connected layers. We add some dropout to the LSTM since even 2 epochs is enough to overfit."},{"metadata":{"trusted":true,"_uuid":"402a9dd0dab55a102580dd3d8e0f9372234022d2"},"cell_type":"code","source":"seed_everything()\ninp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)\nx = Bidirectional(LSTM(50, return_sequences=True, dropout=0.1, recurrent_dropout=0.1))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(50, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d331e20cfcd68246e611e4a0161fb872c3d2112c"},"cell_type":"code","source":"def train_validate_test_split(df, df_y, train_percent=.6, validate_percent=.2, random_state=10):\n    np.random.seed(random_state)\n    perm = np.random.permutation(len(df))\n    m = len(df)\n    train_end = int(train_percent * m)\n    validate_end = int(validate_percent * m) + train_end\n    train = df[perm[:train_end]]\n    validate = df[perm[train_end:validate_end]]\n    test = df[perm[validate_end:]]\n    \n    train_y = df_y[perm[:train_end]]\n    validate_y = df_y[perm[train_end:validate_end]]\n    test_y = df_y[perm[validate_end:]]\n\n    return train, validate, test, train_y, validate_y, test_y, perm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7372b2f834755a19b80703a264d3ab1953a3b287"},"cell_type":"code","source":"# Train / Val / Test -split\nseed_everything()\nX_tra, X_val, X_test, y_tra, y_val, y_test, permutation = train_validate_test_split(train_x, train_y, \n                                train_percent=0.95, validate_percent=0.04, random_state=SEED+2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"53a1c95c754cb41e6ab6a7583b613949bde01fbc"},"cell_type":"code","source":"print(len(X_tra))\nprint(len(X_val))\nprint(len(X_test))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a1f731286990ba5094e846a153186a18d5a17440"},"cell_type":"code","source":"seed_everything()\nmodel.fit(X_tra, y_tra, batch_size=batch_size, epochs=n_epochs, validation_data=(X_val, y_val)); #verbose=2","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9e84488c8547f6a46ed5d27ffc1c12d43ae9d278"},"cell_type":"markdown","source":"#### Find best threshold on Trainin data"},{"metadata":{"trusted":true,"_uuid":"d06f493cf027d8f13d52aecb6f2798d732fbab29"},"cell_type":"code","source":"train_preds = model.predict(X_tra, batch_size=1024)\nprint(len(train_preds))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ff57691f0b1c5bd03c88f77117d8ec5a29350fe9"},"cell_type":"code","source":"# https://www.kaggle.com/ziliwang/baseline-pytorch-bilstm\ndef bestThresshold(train_y,train_preds):\n    tmp = [0,0,0] # idx, cur, max\n    delta = 0\n    for tmp[0] in tqdm(np.arange(0.1, 0.501, 0.01)):\n        tmp[1] = f1_score(train_y, np.array(train_preds)>tmp[0])\n        if tmp[1] > tmp[2]:\n            delta = tmp[0]\n            tmp[2] = tmp[1]\n    print('best threshold is {:.4f} with F1 score: {:.4f}'.format(delta, tmp[2]))\n    return delta\ndelta = bestThresshold(y_tra,train_preds)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"af8e5f3d2c3d6cef166af599b34be21651f3cf0e"},"cell_type":"markdown","source":"#### Measure F1 for Validation data"},{"metadata":{"trusted":true,"_uuid":"bd72d02f4531a64ed629051fafe2b35b00bcbb64"},"cell_type":"code","source":"val_preds = model.predict(X_val, batch_size=1024)\nprint(len(val_preds))\nprint(f1_score(y_val, np.array(val_preds)>delta))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d710e6faf2ee1f7eec5873b45484bc932416a8ad"},"cell_type":"markdown","source":"#### Test data"},{"metadata":{"trusted":true,"_uuid":"8d203714d0442a644a73b864d6198a5f9f7bc530"},"cell_type":"code","source":"test_preds = model.predict(X_test, batch_size=1024)\nprint(len(test_preds))\nprint(f1_score(y_test, np.array(test_preds)>delta))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"353fc5dd4efbac18dc0563cb1767f3c12241e48c"},"cell_type":"code","source":"final_preds = model.predict(test_x, batch_size=1024)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d17dd7b0a92ec98134bf8996fd210edaadf7bed6","scrolled":true,"trusted":true},"cell_type":"code","source":"submission = df_test[['qid']].copy()\nsubmission['prediction'] = (final_preds > delta).astype(int)\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1a07a65e47b5dcefaac040f3f633dc077e8f61e","scrolled":true,"trusted":true},"cell_type":"code","source":"!head submission.csv","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3b49b4237f2af853b2a4421bdb2225496dc3a1dd"},"cell_type":"markdown","source":"### Save for later analysis"},{"metadata":{"trusted":true,"_uuid":"6a9b5b67f69efad42b8f59660ff26d43bd38c80e"},"cell_type":"code","source":"train_preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f570f55e4c00e9aaaf8266bfae22cd6339d2e07f"},"cell_type":"code","source":"# predictions\npredicted = pd.DataFrame(train_preds)\npredicted.columns = ['predicted']\npredicted.to_csv('train_preds.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c7b5317a2a48d27639d0a341ebf9f0ebf19c793"},"cell_type":"code","source":"# save the processed form of train-data\ndf_train_preproc = df_train\ndf_train_preproc.to_csv('df_train_preprocessed.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8401bd2dc771292791a05636f886f2dc6c8c3bea"},"cell_type":"markdown","source":"## Part 2. Analysing Classifier errors - Manual Error Analysis"},{"metadata":{"_uuid":"09758759326448fbbea73bef9bbcd75c339a7b7f"},"cell_type":"markdown","source":"#### Load data back"},{"metadata":{"trusted":true,"_uuid":"d45fd02fb5799cf6fc99fd4e76bfa481786755ae"},"cell_type":"code","source":"df_train = pd.read_csv(\"../input/train.csv\")\ndf_train_preproc = pd.read_csv(\"df_train_preprocessed.csv\")\ntrain_preds = pd.read_csv('train_preds.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f1105fed79fdfa02564d8986fab7c9cfcb78316"},"cell_type":"markdown","source":"Real values of testdata"},{"metadata":{"trusted":true,"_uuid":"2aebca78d7bd7a395de0a64aae235d3c3dd21011"},"cell_type":"code","source":"y_train = y_tra; y_train[0:10]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aaf49e35ad7fb3924858b9231b938f9af1a7a824"},"cell_type":"markdown","source":"Our predicted values"},{"metadata":{"trusted":true,"_uuid":"1f46be323ae9a92aab3cfeade39fa3cd53f83c0b"},"cell_type":"code","source":"train_preds[0:10].T","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8276c3cc164a653bef3d82491727c36dc8fda474"},"cell_type":"markdown","source":"#### Add our predictions along the original data"},{"metadata":{"trusted":true,"_uuid":"ec3f78ffb6d8bae1119b3f0d5fd694169cc6f55b"},"cell_type":"code","source":"combined = pd.concat([df_train, train_preds], axis=1, sort=False)\ncombined.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"57413badc1d5999f1071b1fe1e014993894db867"},"cell_type":"markdown","source":"#### Count the error\nReal value of class is 0 or 1, error is how far our prediction is from this."},{"metadata":{"trusted":true,"_uuid":"98daea577a861034903522ad8c73b9d31c31b3a2"},"cell_type":"code","source":"# SIZE of error - |true class - predicted|\ncombined['error'] = abs(combined['target'] - combined['predicted']) \ncombined.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d878dbaf6d4026eeb9061e068cc2a8b912e9952"},"cell_type":"markdown","source":"#### Question_texts are not printed fully, but shortened. Lets fix that"},{"metadata":{"trusted":true,"_uuid":"26fee8833fa2879b3793e797b85bbf5d335c99b2"},"cell_type":"code","source":"# Display whole text of dataframe field and don't cut it\npd.set_option('display.max_colwidth', -1)    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b170980d6bcbc5a17bc6259a2354972aeb97ec97"},"cell_type":"code","source":"combined.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ece00bbc684d16ff4b22ba802360f64f832a3566"},"cell_type":"markdown","source":"#### Sort the items in decreasing order of error amount"},{"metadata":{"trusted":true,"_uuid":"17c28b92f73f8be73e1375948b9b2a87e0625fff"},"cell_type":"code","source":"# List of biggest errors in decreasing order\nsorted = combined.sort_values(by=['error'], ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"04f51cc659806b3f5f1532deacdaaa694712da2c"},"cell_type":"markdown","source":"### False Positive with biggest error  - Sincere but predicted strongly as insincere"},{"metadata":{"trusted":true,"_uuid":"e13d05dc99ddbe159ca535929540ff16312bfc66"},"cell_type":"code","source":"sorted[ sorted['target']==0] [0:14]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c9960f68b7b72e317a34cc1ca3ae009d876d0029"},"cell_type":"markdown","source":"#### Here it is not instantly obvious, why many of these are predicted as insincere"},{"metadata":{"_uuid":"a17018fcda7da2af1145194531cb304f8361a37d"},"cell_type":"markdown","source":"### View False negatives - Insinsere but predicted as sincere"},{"metadata":{"trusted":true,"_uuid":"1fc7eaa162f31935e7c9d03207ed2b7091eebb34"},"cell_type":"code","source":" pd.options.display.float_format = \"{:.8f}\".format","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"905d8d9bad7460f2ec100ac3957d3d50900dde7c"},"cell_type":"code","source":"# pick texts where true target was 1\ninsincere = sorted[sorted['target']==1]\ninsincere[0:14]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4af15fbe0cc80e189aede238ba5227b6d1d9915c"},"cell_type":"markdown","source":"#### How about most correct predictions?"},{"metadata":{"trusted":true,"_uuid":"97370f32f81b9a5c081657b872eac7b6a2b76faa"},"cell_type":"code","source":"# List of errors in increasing order\nsorted_increasing = combined.sort_values(by=['error'], ascending=True)\nsorted_increasing[0:10]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6c4e9e92d8519a08133f565e1368609c3928aa31"},"cell_type":"markdown","source":"#### Shortest and longest questions"},{"metadata":{"trusted":true,"_uuid":"4147b1a5dd6adf0fbca693ad6fdf8359570c80b6"},"cell_type":"code","source":"# add new filed 'question_length' in characters\ncombined['question_length']=combined['question_text'].apply(lambda x: len(x))\n# sort by that field\nsorted_len = combined.sort_values(by=['question_length'], ascending=True)\nsorted_len[0:20]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a4d433b011bd9a82c6ab8bb6955b7718217dac2"},"cell_type":"markdown","source":"### Here we can see that almost all very short questions are Insinsere, while our model predicts them mostly as sincere!"},{"metadata":{"_uuid":"64f8a190fad496be9858354d136ef5d8a4d96186"},"cell_type":"markdown","source":"#### How about longest questions?"},{"metadata":{"trusted":true,"_uuid":"2c762f0674f4ccaf0db7705fad2cc358b852c0be"},"cell_type":"code","source":"# reverse order - start from longest\n# This print is very wide, so omit printing qid-field\nsorted_len.drop('qid',axis=1).iloc[::-1][0:8]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2fce570801c75d96ebc58518a1589f1e92f4d107"},"cell_type":"markdown","source":"### Strangely many of these big math formulas are labeled as insincere in original data. Evil math?"},{"metadata":{"_uuid":"1b77edcb2bae25b50d3c5af36e606d7eb200cc1b"},"cell_type":"markdown","source":"In one sentence we can also notice that there are http addresses within the data. "},{"metadata":{"trusted":true,"_uuid":"034aedb089ebafd0bf856a0b6da867a1e0690847"},"cell_type":"code","source":"sorted_len[sorted_len['qid']=='4d2e2796dd1ced2c8e64']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3c44d190eed1a8bf2ee2dcbb7abbbbbce88260ca"},"cell_type":"markdown","source":"If you wish to explore more, you can view 100-200 examples of these groups and grasp an understanding of what the data is like and see what you might find."},{"metadata":{"_uuid":"7759ad4a34e659d9beb6532382f36c8fe8816887"},"cell_type":"markdown","source":"One possible way to go further would be to use model with attention layer and visualize the attention - how much does each word in the sentence give weight to chosen class."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.8"}},"nbformat":4,"nbformat_minor":1}