{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"* Họ và tên : Nguyễn Trọng Đạt\n* MSSV : 19021240\n* Lớp : INT3405E_20","metadata":{}},{"cell_type":"markdown","source":"# 1. Mô tả bài toán","metadata":{"papermill":{"duration":0.042128,"end_time":"2022-01-07T01:39:45.064376","exception":false,"start_time":"2022-01-07T01:39:45.022248","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Giới thiệu bài toán\n1. Quora là một nền tảng cho phép mọi người học hỏi lẫn nhau. Trên Quora, mọi người có thể đặt câu hỏi và kết nối với những người khác, những người đóng góp thông tin chi tiết độc đáo và câu trả lời chất lượng.\n2. Một vấn đề tồn tại đối với bất kỳ trang web lớn nào hiện nay là làm thế nào để xử lý nội dung độc hại và gây chia rẽ.\n3. Quora muốn giải quyết vấn đề này trực tiếp để giữ cho nền tảng của họ trở thành một nơi mà người dùng có thể cảm thấy an toàn khi chia sẻ kiến thức của họ với thế giới.\n4. Một thách thức quan trọng là loại bỏ những câu hỏi thiếu chân thành - những câu hỏi được đặt ra dựa trên những tiền đề sai lầm hoặc có ý định đưa ra một tuyên bố hơn là tìm kiếm những câu trả lời hữu ích.\n5. Trong notebook này, đề xuất phát triển các mô hình xác định và gắn cờ cho các \"insincere questions\"\n","metadata":{"papermill":{"duration":0.041153,"end_time":"2022-01-07T01:39:45.147384","exception":false,"start_time":"2022-01-07T01:39:45.106231","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Thêm các thư viện cần thiết","metadata":{"papermill":{"duration":0.039256,"end_time":"2022-01-07T01:39:45.226669","exception":false,"start_time":"2022-01-07T01:39:45.187413","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport random\n%matplotlib inline\nimport matplotlib as mp\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport collections\nfrom wordcloud import WordCloud,STOPWORDS\nimport re\nfrom tqdm import tqdm_notebook, tnrange\nfrom tqdm.auto import tqdm\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nimport torch.nn as nn\nfrom torch.utils.data import Dataset, DataLoader\nimport torch\nimport time\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import f1_score\nfrom nltk.corpus import stopwords\nfrom sklearn.metrics import classification_report\nfrom sklearn.metrics import confusion_matrix\nimport nltk\nfrom nltk.stem import WordNetLemmatizer\nfrom nltk.tokenize.toktok import ToktokTokenizer","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":7.041003,"end_time":"2022-01-07T01:39:52.306839","exception":false,"start_time":"2022-01-07T01:39:45.265836","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:17:23.479952Z","iopub.execute_input":"2022-01-07T17:17:23.480696Z","iopub.status.idle":"2022-01-07T17:17:23.491483Z","shell.execute_reply.started":"2022-01-07T17:17:23.48066Z","shell.execute_reply":"2022-01-07T17:17:23.490727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Tham số ","metadata":{"papermill":{"duration":0.042274,"end_time":"2022-01-07T01:39:52.390885","exception":false,"start_time":"2022-01-07T01:39:52.348611","status":"completed"},"tags":[]}},{"cell_type":"code","source":"embedding_dim = 300 # chiều của vécto embedding từ\nvocab_size = 150000 #số lượng vocab từ lấy ra từ thư viện embedding\nmaxlen = 70 # chiều dài tối đa của 1 từ sau khi làm tròn\n\ntagset_size = 1 # số lượng nhãn\nbatch_size=512\n# Decide which device we want to run on\ndevice = torch.device(\"cuda:0\" if (torch.cuda.is_available()) else \"cpu\")","metadata":{"papermill":{"duration":0.103479,"end_time":"2022-01-07T01:39:52.536","exception":false,"start_time":"2022-01-07T01:39:52.432521","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:17:26.162611Z","iopub.execute_input":"2022-01-07T17:17:26.163359Z","iopub.status.idle":"2022-01-07T17:17:26.170188Z","shell.execute_reply.started":"2022-01-07T17:17:26.163319Z","shell.execute_reply":"2022-01-07T17:17:26.167389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Đảm bảo tính xác định\nVì ở phần sau, em dùng mô hình train trên GPU nên sẽ thiếu tính \"determinism\" sau mỗi lần train. Có thể hiểu nôm na rằng sau 2 lần train với cùng một bộ dữ liệu, chúng ta sẽ có 2 mô hình không hoàn toàn giống nhau, một cái tốt hơn và một cái cho ra kết quả tệ hơn. \\\nMặc dù sự sai khác của 2 mô hình này là không nhiều nhưng để ổn định hơn, em sẽ xử lý nó bằng cách sử dụng \"seed\". Seed là một điểm bắt đầu trong một chuỗi xác định, nó đảm bảo rằng khi ta dùng cùng một seed, kết quả cho ra qua các lần chạy đều giống nhau","metadata":{}},{"cell_type":"code","source":"def set_seed(seed_value=2022):\n \"\"\"Set seed for reproducibility.\"\"\"\n\n random.seed(seed_value)\n np.random.seed(seed_value)\n torch.manual_seed(seed_value)\n torch.cuda.manual_seed_all(seed_value)\n torch.backends.cudnn.deterministic = True\nset_seed()","metadata":{"execution":{"iopub.status.busy":"2022-01-07T17:17:27.806451Z","iopub.execute_input":"2022-01-07T17:17:27.806707Z","iopub.status.idle":"2022-01-07T17:17:27.815271Z","shell.execute_reply.started":"2022-01-07T17:17:27.806678Z","shell.execute_reply":"2022-01-07T17:17:27.81442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Phân tích dữ liệu\n","metadata":{"papermill":{"duration":0.039823,"end_time":"2022-01-07T01:39:52.616214","exception":false,"start_time":"2022-01-07T01:39:52.576391","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## 2.1 Khảo sát dữ liệu\n","metadata":{"papermill":{"duration":0.039071,"end_time":"2022-01-07T01:39:52.69521","exception":false,"start_time":"2022-01-07T01:39:52.656139","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### 2.1.1Đọc dữ liệu","metadata":{"papermill":{"duration":0.039014,"end_time":"2022-01-07T01:39:52.773943","exception":false,"start_time":"2022-01-07T01:39:52.734929","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df = pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ntest_df = pd.read_csv('../input/quora-insincere-questions-classification/test.csv')","metadata":{"papermill":{"duration":5.645325,"end_time":"2022-01-07T01:39:58.458476","exception":false,"start_time":"2022-01-07T01:39:52.813151","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:07:28.185525Z","iopub.execute_input":"2022-01-07T17:07:28.185779Z","iopub.status.idle":"2022-01-07T17:07:33.673503Z","shell.execute_reply.started":"2022-01-07T17:07:28.185747Z","shell.execute_reply":"2022-01-07T17:07:33.672733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.1.2 Tổng quan về dữ liệu","metadata":{"papermill":{"duration":0.039156,"end_time":"2022-01-07T01:39:58.537373","exception":false,"start_time":"2022-01-07T01:39:58.498217","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df.head()","metadata":{"papermill":{"duration":0.095229,"end_time":"2022-01-07T01:39:58.672104","exception":false,"start_time":"2022-01-07T01:39:58.576875","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:07:33.675881Z","iopub.execute_input":"2022-01-07T17:07:33.676394Z","iopub.status.idle":"2022-01-07T17:07:33.697801Z","shell.execute_reply.started":"2022-01-07T17:07:33.676351Z","shell.execute_reply":"2022-01-07T17:07:33.697112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.tail()","metadata":{"papermill":{"duration":0.105003,"end_time":"2022-01-07T01:39:58.857035","exception":false,"start_time":"2022-01-07T01:39:58.752032","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:07:33.700761Z","iopub.execute_input":"2022-01-07T17:07:33.701041Z","iopub.status.idle":"2022-01-07T17:07:33.711562Z","shell.execute_reply.started":"2022-01-07T17:07:33.701013Z","shell.execute_reply":"2022-01-07T17:07:33.710704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Bộ dữ liệu có 3 cột là **qid** - id của câu hỏi, **question_text** - nội dung của câu hỏi và **target** - phân loại câu hỏi bằng hai giá trị là 0, 1 với 0 là các câu hỏi \"**sincere**\" và 1 là các câu hỏi \"**insincere**\"","metadata":{"papermill":{"duration":0.068296,"end_time":"2022-01-07T01:39:58.997341","exception":false,"start_time":"2022-01-07T01:39:58.929045","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df.info()","metadata":{"papermill":{"duration":0.358407,"end_time":"2022-01-07T01:39:59.42326","exception":false,"start_time":"2022-01-07T01:39:59.064853","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:07:33.713894Z","iopub.execute_input":"2022-01-07T17:07:33.714284Z","iopub.status.idle":"2022-01-07T17:07:33.984463Z","shell.execute_reply.started":"2022-01-07T17:07:33.714126Z","shell.execute_reply":"2022-01-07T17:07:33.98335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Bộ dữ liệu có 1306122 câu hỏi \\\nCác câu hỏi \"sincere\"","metadata":{"papermill":{"duration":0.040719,"end_time":"2022-01-07T01:39:59.504738","exception":false,"start_time":"2022-01-07T01:39:59.464019","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df[train_df['target']==0].head()","metadata":{"papermill":{"duration":0.121691,"end_time":"2022-01-07T01:39:59.666991","exception":false,"start_time":"2022-01-07T01:39:59.5453","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:07:33.986757Z","iopub.execute_input":"2022-01-07T17:07:33.987021Z","iopub.status.idle":"2022-01-07T17:07:34.079395Z","shell.execute_reply.started":"2022-01-07T17:07:33.986985Z","shell.execute_reply":"2022-01-07T17:07:34.078726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Các câu hỏi \"insincere\"","metadata":{"papermill":{"duration":0.040632,"end_time":"2022-01-07T01:39:59.748887","exception":false,"start_time":"2022-01-07T01:39:59.708255","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df[train_df['target']==1].head()","metadata":{"papermill":{"duration":0.067597,"end_time":"2022-01-07T01:39:59.857256","exception":false,"start_time":"2022-01-07T01:39:59.789659","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:07:34.080802Z","iopub.execute_input":"2022-01-07T17:07:34.081237Z","iopub.status.idle":"2022-01-07T17:07:34.108152Z","shell.execute_reply.started":"2022-01-07T17:07:34.081199Z","shell.execute_reply":"2022-01-07T17:07:34.107503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2 Visualize data\n","metadata":{"papermill":{"duration":0.044353,"end_time":"2022-01-07T01:39:59.943708","exception":false,"start_time":"2022-01-07T01:39:59.899355","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### 2.2.1 Thống kê về dữ liệu","metadata":{"papermill":{"duration":0.049403,"end_time":"2022-01-07T01:40:00.037431","exception":false,"start_time":"2022-01-07T01:39:59.988028","status":"completed"},"tags":[]}},{"cell_type":"code","source":"labels = collections.Counter(train_df['target']).keys()\namounts = collections.Counter(train_df['target']).values()\nlabels, amounts","metadata":{"papermill":{"duration":0.435106,"end_time":"2022-01-07T01:40:00.520302","exception":false,"start_time":"2022-01-07T01:40:00.085196","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:07:34.109498Z","iopub.execute_input":"2022-01-07T17:07:34.109772Z","iopub.status.idle":"2022-01-07T17:07:34.50863Z","shell.execute_reply.started":"2022-01-07T17:07:34.109738Z","shell.execute_reply":"2022-01-07T17:07:34.507886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"insincere_data_train = train_df[train_df.target == 1]\nsincere_data_train = train_df[train_df.target == 0]\nprint(insincere_data_train.shape, sincere_data_train.shape)\nprint(\"--------------------------\")\n\nsns.countplot(x='target', data=train_df)","metadata":{"papermill":{"duration":0.551162,"end_time":"2022-01-07T01:40:01.113504","exception":false,"start_time":"2022-01-07T01:40:00.562342","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:07:34.509974Z","iopub.execute_input":"2022-01-07T17:07:34.510916Z","iopub.status.idle":"2022-01-07T17:07:34.854376Z","shell.execute_reply.started":"2022-01-07T17:07:34.510866Z","shell.execute_reply":"2022-01-07T17:07:34.853692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sincere_percent= (len(train_df.question_text[train_df['target'] == 0]) / len(train_df['question_text']) * 100)\ninsincere_percent= (len(train_df.question_text[train_df['target'] == 1]) / len(train_df['question_text']) * 100)\n__labels = 'sincere', 'insincere'\nsizes = [sincere_percent, insincere_percent]\nexplode = (0.1, 0) # explode 1st slice\n\nplt.pie(sizes, explode=explode, labels=__labels,autopct='%1.1f%%', shadow=True, startangle=140)\n\nplt.axis('equal')\nplt.show()","metadata":{"papermill":{"duration":0.193601,"end_time":"2022-01-07T01:40:01.350638","exception":false,"start_time":"2022-01-07T01:40:01.157037","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:07:34.855536Z","iopub.execute_input":"2022-01-07T17:07:34.855918Z","iopub.status.idle":"2022-01-07T17:07:35.00819Z","shell.execute_reply.started":"2022-01-07T17:07:34.855878Z","shell.execute_reply":"2022-01-07T17:07:35.007522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nhận thấy **\"sincere question\"** có hơn 1,2 triệu câu hỏi, chiếm 93.8% tập train, còn lại 6.2% cho **\"insincere question\"** với khoảng gần 81,000 câu hỏi. Nhận thấy lượng data có sự chênh lệnh lớn khi số lượng **\"sincere question\"** gấp 15 lần so với **\"insincere question\"**. Với lượng data bị mất cân bằng giữa các nhãn như thế này, chúng ta cần phải có một biện pháp khắc phục sự mất cân bằng để mô hình cho được kết quả tốt nhất.","metadata":{"papermill":{"duration":0.043141,"end_time":"2022-01-07T01:40:01.43709","exception":false,"start_time":"2022-01-07T01:40:01.393949","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### 2.2.1 Phân tích câu trong data\nThống kê số kí tự trong từng câu","metadata":{"papermill":{"duration":0.043875,"end_time":"2022-01-07T01:40:01.524602","exception":false,"start_time":"2022-01-07T01:40:01.480727","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df['question_text'].str.len().hist()","metadata":{"papermill":{"duration":1.114192,"end_time":"2022-01-07T01:40:02.682639","exception":false,"start_time":"2022-01-07T01:40:01.568447","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:07:35.010636Z","iopub.execute_input":"2022-01-07T17:07:35.011059Z","iopub.status.idle":"2022-01-07T17:07:36.080477Z","shell.execute_reply.started":"2022-01-07T17:07:35.01102Z","shell.execute_reply":"2022-01-07T17:07:36.079688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Số lượng từ có trong 1 câu","metadata":{"papermill":{"duration":0.052301,"end_time":"2022-01-07T01:40:02.786086","exception":false,"start_time":"2022-01-07T01:40:02.733785","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df['question_text'].str.split().map(lambda x: len(x)).hist()","metadata":{"papermill":{"duration":5.168622,"end_time":"2022-01-07T01:40:08.005786","exception":false,"start_time":"2022-01-07T01:40:02.837164","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:07:36.081584Z","iopub.execute_input":"2022-01-07T17:07:36.081821Z","iopub.status.idle":"2022-01-07T17:07:41.752433Z","shell.execute_reply.started":"2022-01-07T17:07:36.081786Z","shell.execute_reply":"2022-01-07T17:07:41.751757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Chúng ta có thể thấy độ dài các từ trong khoảng 0-20 chiếm phần đa dữ liệu . Max độ dài từ cũng chỉ là 70. Từ đó chúng ta chọn chiều dài biểu diễn vecto trong câu là 70\\\nmaxlen = 70 # max number of words in a question to use","metadata":{"papermill":{"duration":0.045743,"end_time":"2022-01-07T01:40:08.096746","exception":false,"start_time":"2022-01-07T01:40:08.051003","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### Chúng ta cùng xem qua các từ xuất hiện nhiều nhất trong các câu hỏi của dữ liệu\n","metadata":{"papermill":{"duration":0.045074,"end_time":"2022-01-07T01:40:08.187076","exception":false,"start_time":"2022-01-07T01:40:08.142002","status":"completed"},"tags":[]}},{"cell_type":"code","source":"new_corpus = []\nquest = train_df['question_text'].str.split()\nquest = quest.values.tolist()\nnew_corpus = [word for q in quest for word in q]\ncounter = collections.Counter(new_corpus)\nmost = counter.most_common()\n\nx,y= [],[]\nfor word,count in most[:20]:\n x.append(word)\n y.append(count)\n\nsns.barplot(x=y,y=x).set(title='Các từ xuất hiện nhiều nhất trong dữ liệu Train')","metadata":{"papermill":{"duration":8.486615,"end_time":"2022-01-07T01:40:16.719343","exception":false,"start_time":"2022-01-07T01:40:08.232728","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:07:41.753747Z","iopub.execute_input":"2022-01-07T17:07:41.754145Z","iopub.status.idle":"2022-01-07T17:07:49.011029Z","shell.execute_reply.started":"2022-01-07T17:07:41.754106Z","shell.execute_reply":"2022-01-07T17:07:49.010369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_corpus = []\nquest = test_df['question_text'].str.split()\nquest = quest.values.tolist()\nnew_corpus = [word for q in quest for word in q]\ncounter = collections.Counter(new_corpus)\nmost = counter.most_common()\n\nx,y= [],[]\nfor word,count in most[:20]:\n x.append(word)\n y.append(count)\n\nsns.barplot(x=y,y=x).set(title='Các từ xuất hiện nhiều nhất trong dữ liệu Test')","metadata":{"execution":{"iopub.status.busy":"2022-01-07T17:07:49.012215Z","iopub.execute_input":"2022-01-07T17:07:49.012455Z","iopub.status.idle":"2022-01-07T17:07:52.73246Z","shell.execute_reply.started":"2022-01-07T17:07:49.012421Z","shell.execute_reply":"2022-01-07T17:07:52.731716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Xem các cụm từ xuất hiện nhiều trong các câu hỏi","metadata":{}},{"cell_type":"code","source":"def bigrams_data(data):\n bigrams = []\n for question in data:\n question = [w for w in question.lower().split() if w not in STOPWORDS] # skip từ stopword\n if not question: \n continue # skip question do nltk.bigrmas\n bi = [b for b in nltk.bigrams(question)]\n bi = [' '.join(w) for w in bi]\n bigrams.extend(bi)\n return bigrams\n\ndef draw_plt(data, title, bar_color, numberOfWordsInTop):\n top_words = collections.Counter(data).most_common(numberOfWordsInTop) # 25 từ xuất hiện nhiều nhất\n\n df_top = pd.DataFrame(top_words, columns=['word', 'count']).sort_values('count')\n\n plt.barh(df_top['word'].values, df_top['count'].values, orientation='horizontal', color=bar_color)\n plt.title(f'Top words in {title}')\n\nbigrams_sincere = bigrams_data(train_df[train_df['target']==0]['question_text'])\nbigrams_insincere = bigrams_data(train_df[train_df['target']==1]['question_text'])","metadata":{"execution":{"iopub.status.busy":"2022-01-07T17:07:52.73392Z","iopub.execute_input":"2022-01-07T17:07:52.734788Z","iopub.status.idle":"2022-01-07T17:08:03.503565Z","shell.execute_reply.started":"2022-01-07T17:07:52.734746Z","shell.execute_reply":"2022-01-07T17:08:03.502823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.subplot(1, 2, 1)\ndraw_plt(bigrams_sincere, 'Các cụm từ xuất hiện nhiều nhất trong câu sincere ', 'blue',25)\n\n\nplt.subplot(1, 2, 2)\ndraw_plt(bigrams_insincere, 'Các cụm từ xuất hiện nhiều nhất trong câu insincere', 'red',25)\n\nplt.subplots_adjust(right=3.0)\nplt.subplots_adjust(top=2.0)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-07T17:08:03.504791Z","iopub.execute_input":"2022-01-07T17:08:03.505062Z","iopub.status.idle":"2022-01-07T17:08:07.59982Z","shell.execute_reply.started":"2022-01-07T17:08:03.505028Z","shell.execute_reply":"2022-01-07T17:08:07.599147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Khảo sát các từ xuất hiện nhiều trong các câu hỏi\nBiểu đồ word cloud","metadata":{"papermill":{"duration":0.049352,"end_time":"2022-01-07T01:40:16.821716","exception":false,"start_time":"2022-01-07T01:40:16.772364","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def cloud(docs, title):\n wordcloud = WordCloud(width=800, height=400, collocations=False, background_color=\"white\").generate(\" \".join(docs))\n fig = plt.figure(figsize=(10,7), facecolor='w')\n plt.imshow(wordcloud)\n plt.axis('off')\n plt.title(title, fontsize=25,color='k')\n plt.tight_layout(pad=0)\n plt.show()\ncloud(train_df[train_df['target']==0]['question_text'], \"Sincere question\")\ncloud(test_df['question_text'][train_df['target']==1], \"Insincere question\")","metadata":{"papermill":{"duration":22.769785,"end_time":"2022-01-07T01:40:39.64269","exception":false,"start_time":"2022-01-07T01:40:16.872905","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:08:07.602458Z","iopub.execute_input":"2022-01-07T17:08:07.602903Z","iopub.status.idle":"2022-01-07T17:08:29.666293Z","shell.execute_reply.started":"2022-01-07T17:08:07.602864Z","shell.execute_reply":"2022-01-07T17:08:29.663033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.3 Tiền xử lý dữ liệu\nTham khảo tại : https://www.kaggle.com/sunnymarkliu/more-text-cleaning-to-increase-word-coverage\n","metadata":{"papermill":{"duration":0.060633,"end_time":"2022-01-07T01:40:39.765112","exception":false,"start_time":"2022-01-07T01:40:39.704479","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### 2.3.1 Làm sạch dữ liệu\n**clean_text** :xử lý các ký tự đặc biệt tồn tại trong văn bản đầu vào, các ký tự đặc biệt được lấy từ mảng puncts bên dưới. Hiểu đơn giản là bỏ đi các ký tự đặc biệt và xóa dấu câu\\\n**clean_number** : Xử lý các chữ số đầu vào, thay thế bằng các ký tự #s tại vì các thư viện embedding đã xử lý các số theo cách này\\\n**replace_typical_misspell**: xử lý các từ viết tắt trong câu đầu vào bằng **mispell_dict** tương ứng","metadata":{"papermill":{"duration":0.060112,"end_time":"2022-01-07T01:40:39.886043","exception":false,"start_time":"2022-01-07T01:40:39.825931","status":"completed"},"tags":[]}},{"cell_type":"code","source":"puncts = [',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', \"'\", '$', '&', '/', '[', ']', '>', '%', '=', '#', '*', '+', '\\\\', '•', '~', '@', '£', \n '·', '_', '{', '}', '©', '^', '®', '`', '<', '→', '°', '€', '™', '›', '♥', '←', '×', '§', '″', '′', 'Â', '█', '½', 'à', '…', \n '“', '★', '”', '–', '●', 'â', '►', '−', '¢', '²', '¬', '░', '¶', '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', \n '▒', '：', '¼', '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲', 'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', \n '∙', '）', '↓', '、', '│', '（', '»', '，', '♪', '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø', '¹', '≤', '‡', '√', ]\npunct_mapping = {\"‘\": \"'\", \"₹\": \"e\", \"´\": \"'\", \"°\": \"\", \"€\": \"e\", \"™\": \"tm\", \"√\": \" sqrt \", \"×\": \"x\", \"²\": \"2\",\n \"—\": \"-\", \"–\": \"-\", \"’\": \"'\", \"_\": \"-\", \"`\": \"'\", '”': '\"', '“': '\"', \"£\": \"e\",\n '∞': 'infinity', 'θ': 'theta', '÷': '/', 'α': 'alpha', '•': '.', 'à': 'a', '−': '-', 'β': 'beta',\n '∅': '', '³': '3', 'π': 'pi', '\\u200b': ' ', '…': ' ... ', '\\ufeff': '', 'करना': '', 'है': ''}\nfor p in puncts:\n punct_mapping[p] = ' %s ' % p\n\np = re.compile('(\\[ math \\]).+(\\[ / math \\])')\np_space = re.compile(r'[^\\x20-\\x7e]')\n\n#\ndef remove_stopwords(text):\n text = [word for word in text.split() if word not in STOPWORDS]\n text = ' '.join(text)\n return text\n\ndef clean_text(text):\n # clean latex maths\n text = p.sub(' [ math ] ', text)\n # clean invisible chars\n text = p_space.sub(r'', text)\n # clean punctuations\n for punct in punct_mapping:\n if punct in text:\n text = text.replace(punct, punct_mapping[punct])\n #Remove stop word\n# text = remove_stopwords(text)\n return text\n\ndef clean_numbers(x):\n if bool(re.search(r'\\d', x)):\n x = re.sub('[0-9]{5,}', '#####', x)\n x = re.sub('[0-9]{4}', '####', x)\n x = re.sub('[0-9]{3}', '###', x)\n x = re.sub('[0-9]{2}', '##', x)\n return x\n\nmispell_dict = {\"ain't\": \"is not\", \"aren't\": \"are not\", \"can't\": \"cannot\", \"'cause\": \"because\",\n \"could've\": \"could have\", \"couldn't\": \"could not\", \"didn't\": \"did not\", \"doesn't\": \"does not\",\n \"don't\": \"do not\", \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\",\n \"he'd\": \"he would\", \"he'll\": \"he will\", \"he's\": \"he is\", \"how'd\": \"how did\",\n \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\", \"i'd\": \"i would\",\n \"i'd've\": \"i would have\", \"i'll\": \"i will\", \"i'll've\": \"I will have\", \"i'm\": \"i am\",\n \"i've\": \"I have\", \"isn't\": \"is not\", \"it'd\": \"it would\",\n \"it'd've\": \"it would have\", \"it'll\": \"it will\", \"it'll've\": \"it will have\", \"it's\": \"it is\",\n \"let's\": \"let us\", \"ma'am\": \"madam\", \"mayn't\": \"may not\", \"might've\": \"might have\",\n \"mightn't\": \"might not\", \"mightn't've\": \"might not have\", \"must've\": \"must have\",\n \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\",\n \"needn't've\": \"need not have\", \"o'clock\": \"of the clock\", \"oughtn't\": \"ought not\",\n \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\", \"sha'n't\": \"shall not\",\n \"shan't've\": \"shall not have\", \"she'd\": \"she would\", \"she'd've\": \"she would have\",\n \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\",\n \"should've\": \"should have\", \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\",\n \"so've\": \"so have\", \"so's\": \"so as\", \"this's\": \"this is\", \"that'd\": \"that would\",\n \"that'd've\": \"that would have\", \"that's\": \"that is\", \"there'd\": \"there would\",\n \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\n \"they'd\": \"they would\", \"they'd've\": \"they would have\", \"they'll\": \"they will\",\n \"they'll've\": \"they will have\", \"they're\": \"they are\", \"they've\": \"they have\",\n \"to've\": \"to have\", \"wasn't\": \"was not\", \"we'd\": \"we would\", \"we'd've\": \"we would have\",\n \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\", \"we've\": \"we have\",\n \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\",\n \"what're\": \"what are\", \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\",\n \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\", \"where've\": \"where have\",\n \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\",\n \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\",\n \"won't've\": \"will not have\", \"would've\": \"would have\", \"wouldn't\": \"would not\",\n \"wouldn't've\": \"would not have\", \"y'all\": \"you all\", \"y'all'd\": \"you all would\",\n \"y'all'd've\": \"you all would have\", \"y'all're\": \"you all are\", \"y'all've\": \"you all have\",\n \"you'd\": \"you would\", \"you'd've\": \"you would have\", \"you'll\": \"you will\",\n \"you'll've\": \"you will have\", \"you're\": \"you are\", \"you've\": \"you have\", 'colour': 'color',\n 'centre': 'center', 'favourite': 'favorite', 'travelling': 'traveling',\n 'counselling': 'counseling', 'theatre': 'theater', 'cancelled': 'canceled', 'labour': 'labor',\n 'organisation': 'organization', 'wwii': 'world war 2', 'citicise': 'criticize',\n 'youtu ': 'youtube ', 'qoura': 'quora', 'sallary': 'salary', 'whta': 'what',\n 'narcisist': 'narcissist', 'howdo': 'how do', 'whatare': 'what are', 'howcan': 'how can',\n 'howmuch': 'how much', 'howmany': 'how many', 'whydo': 'why do', 'doi': 'do I',\n 'thebest': 'the best', 'howdoes': 'how does', 'mastrubation': 'masturbation',\n 'mastrubate': 'masturbate', \"mastrubating\": 'masturbating', 'pennis': 'penis',\n 'etherium': 'ethereum', 'narcissit': 'narcissist', 'bigdata': 'big data', '2k17': '2017',\n '2k18': '2018', 'qouta': 'quota', 'exboyfriend': 'ex boyfriend', 'airhostess': 'air hostess',\n \"whst\": 'what', 'watsapp': 'whatsapp', 'demonitisation': 'demonetization',\n 'demonitization': 'demonetization', 'demonetisation': 'demonetization'}\n#Removing Contractions/ chữa các từ chính tả\ndef replace_typical_misspell(text):\n tokens = []\n for token in text.split():\n # replace contractions & correct misspells\n token = mispell_dict.get(token.lower(), token)\n tokens.append(token)\n text = ' '.join(tokens)\n return text\n \n#Remove contraction / Xoá các từ contraction\ncontraction_map = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \"'cause\": \"because\", \"could've\": \"could have\", \"couldn't\": \"could not\", \"didn't\": \"did not\", \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\", \"he'd\": \"he would\",\"he'll\": \"he will\", \"he's\": \"he is\", \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\", \"I'd\": \"I would\", \"I'd've\": \"I would have\", \"I'll\": \"I will\", \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\", \"i'd\": \"i would\", \"i'd've\": \"i would have\", \"i'll\": \"i will\", \"i'll've\": \"i will have\",\"i'm\": \"i am\", \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\", \"it'd've\": \"it would have\", \"it'll\": \"it will\", \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \"ma'am\": \"madam\", \"mayn't\": \"may not\", \"might've\": \"might have\",\"mightn't\": \"might not\",\"mightn't've\": \"might not have\", \"must've\": \"must have\", \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\"o'clock\": \"of the clock\", \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\", \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\", \"she'd\": \"she would\", \"she'd've\": \"she would have\", \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\", \"should've\": \"should have\", \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\"so's\": \"so as\", \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \"that's\": \"that is\", \"there'd\": \"there would\", \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\"they'd\": \"they would\", \"they'd've\": \"they would have\", \"they'll\": \"they will\", \"they'll've\": \"they will have\", \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\", \"wasn't\": \"was not\", \"we'd\": \"we would\", \"we'd've\": \"we would have\", \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\", \"we've\": \"we have\", \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\", \"what're\": \"what are\", \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\", \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\", \"where've\": \"where have\", \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\", \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\", \"won't've\": \"will not have\", \"would've\": \"would have\", \"wouldn't\": \"would not\", \"wouldn't've\": \"would not have\", \"y'all\": \"you all\", \"y'all'd\": \"you all would\",\"y'all'd've\": \"you all would have\",\"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\"you'd\": \"you would\", \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\", \"you're\": \"you are\", \"you've\": \"you have\" }\ndef clean_contractions(text):\n specials = [\"’\", \"‘\", \"´\", \"`\"]\n for s in specials:\n text = text.replace(s, \"'\")\n \n text = ' '.join([contraction_map[t] if t in contraction_map else t for t in text.split(\" \")])\n return text\n ","metadata":{"papermill":{"duration":0.09469,"end_time":"2022-01-07T01:40:40.042166","exception":false,"start_time":"2022-01-07T01:40:39.947476","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:08:29.668202Z","iopub.execute_input":"2022-01-07T17:08:29.668444Z","iopub.status.idle":"2022-01-07T17:08:29.709638Z","shell.execute_reply.started":"2022-01-07T17:08:29.668413Z","shell.execute_reply":"2022-01-07T17:08:29.70888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.3.2 Chuẩn hoá dữ liệu\n1. Chuyển hết tất cả các ký tự trong câu hỏi về dạng viết thường\n2. Xử các ký tự đặc biệt khỏi từng câu\n3. Xử lý các chữ số nằm trong từng câu hỏi\n4. Thay thế các từ viết tắt thành dạng nguyên bản của chúng\n5. Thay thế các giá trị null trong cột questiontext bằng giá trị\n6. Thêm một số trường dữ liệu cho dataset\n7. Tokenize train data và test data : Chuyển text -> ma trận từ\n\n\\\n\n\n8. Padding data : Các mô hình yêu cầu cùng kích thước đầu vào -> Lấp đầy những câu có ít từ (ma trận ngắn)\n\n\n\n\n\n\n\n\n9. Shuffle data: trộn data ngẫu nhiên\n\n","metadata":{"papermill":{"duration":0.066433,"end_time":"2022-01-07T01:40:40.169825","exception":false,"start_time":"2022-01-07T01:40:40.103392","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\ntqdm.pandas()\n# Thêm một số trường dữ liệu\ndef add_features(df):\n \n df['question_text'] = df['question_text'].progress_apply(lambda x:str(x))\n df['total_length'] = df['question_text'].progress_apply(len)\n df['capitals'] = df['question_text'].progress_apply(lambda comment: sum(1 for c in comment if c.isupper()))\n df['caps_vs_length'] = df.progress_apply(lambda row: float(row['capitals'])/float(row['total_length']),\n axis=1)\n df['num_words'] = df.question_text.str.count('\\S+')\n df['num_unique_words'] = df['question_text'].progress_apply(lambda comment: len(set(w for w in comment.split())))\n df['words_vs_unique'] = df['num_unique_words'] / df['num_words'] \n\n return df\n\ndef load_and_prec():\n print(\"Train shape : \",train_df.shape)\n print(\"Test shape : \",test_df.shape)\n \n \n # Lower\n train_df[\"question_text\"] = train_df[\"question_text\"].apply(lambda x: x.lower())\n test_df[\"question_text\"] = test_df[\"question_text\"].apply(lambda x: x.lower())\n\n # Clean the text\n train_df[\"question_text\"] = train_df[\"question_text\"].progress_apply(lambda x: clean_text(x))\n test_df[\"question_text\"] = test_df[\"question_text\"].apply(lambda x: clean_text(x))\n # Clean numbers\n train_df[\"question_text\"] = train_df[\"question_text\"].progress_apply(lambda x: clean_numbers(x))\n test_df[\"question_text\"] = test_df[\"question_text\"].apply(lambda x: clean_numbers(x))\n \n # Clean speelings\n train_df[\"question_text\"] = train_df[\"question_text\"].progress_apply(lambda x: replace_typical_misspell(x))\n test_df[\"question_text\"] = test_df[\"question_text\"].apply(lambda x: replace_typical_misspell(x))\n \n # Fill up the missing values\n train_X = train_df[\"question_text\"].fillna(\"_##_\").values\n test_X = test_df[\"question_text\"].fillna(\"_##_\").values\n \n \n ###################### Add Features ###############################\n train = add_features(train_df)\n test = add_features(test_df)\n \n features = train[['caps_vs_length', 'words_vs_unique']].fillna(0)\n test_features = test[['caps_vs_length', 'words_vs_unique']].fillna(0)\n\n ss = StandardScaler()\n ss.fit(np.vstack((features, test_features)))\n features = ss.transform(features)\n test_features = ss.transform(test_features)\n ###########################################################################\n\n ## Tokenize the sentences\n tokenizer = Tokenizer(num_words=vocab_size)\n tokenizer.fit_on_texts(list(train_X))\n train_X = tokenizer.texts_to_sequences(train_X)\n test_X = tokenizer.texts_to_sequences(test_X)\n\n ## Pad the sentences \n train_X = pad_sequences(train_X, maxlen=maxlen)\n test_X = pad_sequences(test_X, maxlen=maxlen)\n\n ## Get the target values\n train_y = train_df['target'].values\n \n\n #shuffling the data\n np.random.seed(2022)\n trn_idx = np.random.permutation(len(train_X))\n\n train_X = train_X[trn_idx]\n train_y = train_y[trn_idx]\n features = features[trn_idx]\n \n return train_X, test_X, train_y, features, test_features, tokenizer.word_index","metadata":{"papermill":{"duration":0.088143,"end_time":"2022-01-07T01:40:40.326895","exception":false,"start_time":"2022-01-07T01:40:40.238752","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:08:29.711024Z","iopub.execute_input":"2022-01-07T17:08:29.711284Z","iopub.status.idle":"2022-01-07T17:08:29.7291Z","shell.execute_reply.started":"2022-01-07T17:08:29.711244Z","shell.execute_reply":"2022-01-07T17:08:29.728427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train, x_test, y_train,features, test_features, word_index = load_and_prec() ","metadata":{"papermill":{"duration":177.911406,"end_time":"2022-01-07T01:43:38.299467","exception":false,"start_time":"2022-01-07T01:40:40.388061","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:08:29.730337Z","iopub.execute_input":"2022-01-07T17:08:29.730721Z","iopub.status.idle":"2022-01-07T17:11:25.763605Z","shell.execute_reply.started":"2022-01-07T17:08:29.730683Z","shell.execute_reply":"2022-01-07T17:11:25.762789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train","metadata":{"papermill":{"duration":0.077583,"end_time":"2022-01-07T01:43:38.444763","exception":false,"start_time":"2022-01-07T01:43:38.36718","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:11:25.765065Z","iopub.execute_input":"2022-01-07T17:11:25.765309Z","iopub.status.idle":"2022-01-07T17:11:25.773088Z","shell.execute_reply.started":"2022-01-07T17:11:25.765276Z","shell.execute_reply":"2022-01-07T17:11:25.772258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.3.3 Embedding\n\nTrong notebook này em sử dụng 3 thư viện embedding từ có sẵn của Google-glove, Facebook-fasttext, paragram\nTham khảo : https://www.kaggle.com/suicaokhoailang/blending-with-linear-regression-0-688-lb/notebook","metadata":{"papermill":{"duration":0.066714,"end_time":"2022-01-07T01:43:38.577757","exception":false,"start_time":"2022-01-07T01:43:38.511043","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!unzip ../input/quora-insincere-questions-classification/embeddings.zip","metadata":{"papermill":{"duration":219.39979,"end_time":"2022-01-07T01:47:18.084217","exception":false,"start_time":"2022-01-07T01:43:38.684427","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:11:25.774531Z","iopub.execute_input":"2022-01-07T17:11:25.774838Z","iopub.status.idle":"2022-01-07T17:15:01.328841Z","shell.execute_reply.started":"2022-01-07T17:11:25.774801Z","shell.execute_reply":"2022-01-07T17:15:01.328011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_glove(word_index):\n EMBEDDING_FILE = './glove.840B.300d/glove.840B.300d.txt'\n def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n\n all_embs = np.stack(embeddings_index.values())\n emb_mean,emb_std = all_embs.mean(), all_embs.std()\n embed_size = all_embs.shape[1]\n\n # word_index = tokenizer.word_index\n nb_words = min(vocab_size, len(word_index))\n embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n for word, i in tqdm(word_index.items()):\n if i >= vocab_size: continue\n embedding_vector = embeddings_index.get(word)\n if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n \n return embedding_matrix \n \ndef load_fasttext(word_index): \n EMBEDDING_FILE = './wiki-news-300d-1M/wiki-news-300d-1M.vec'\n def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE) if len(o)>100)\n\n all_embs = np.stack(embeddings_index.values())\n emb_mean,emb_std = all_embs.mean(), all_embs.std()\n embed_size = all_embs.shape[1]\n # word_index = tokenizer.word_index\n nb_words = min(vocab_size, len(word_index))\n embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n for word, i in tqdm(word_index.items()):\n if i >= vocab_size: continue\n embedding_vector = embeddings_index.get(word)\n if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n\n return embedding_matrix\n\ndef load_para(word_index):\n EMBEDDING_FILE = './paragram_300_sl999/paragram_300_sl999.txt'\n def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE, encoding=\"utf8\", errors='ignore') if len(o)>100)\n\n all_embs = np.stack(embeddings_index.values())\n emb_mean,emb_std = all_embs.mean(), all_embs.std()\n embed_size = all_embs.shape[1]\n\n # word_index = tokenizer.word_index\n nb_words = min(vocab_size, len(word_index))\n embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n for word, i in tqdm(word_index.items()):\n if i >= vocab_size: continue\n embedding_vector = embeddings_index.get(word)\n if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n \n return embedding_matrix","metadata":{"papermill":{"duration":0.260343,"end_time":"2022-01-07T01:47:18.501221","exception":false,"start_time":"2022-01-07T01:47:18.240878","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:17:36.070116Z","iopub.execute_input":"2022-01-07T17:17:36.070367Z","iopub.status.idle":"2022-01-07T17:17:36.08653Z","shell.execute_reply.started":"2022-01-07T17:17:36.070338Z","shell.execute_reply":"2022-01-07T17:17:36.085795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"set_seed()\nembedding_matrix_1 = load_glove(word_index)\nembedding_matrix_2 = load_fasttext(word_index)\nembedding_matrix_3 = load_para(word_index)","metadata":{"papermill":{"duration":570.595788,"end_time":"2022-01-07T01:56:58.820894","exception":false,"start_time":"2022-01-07T01:47:28.225106","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:17:37.668588Z","iopub.execute_input":"2022-01-07T17:17:37.668863Z","iopub.status.idle":"2022-01-07T17:26:56.005501Z","shell.execute_reply.started":"2022-01-07T17:17:37.668815Z","shell.execute_reply":"2022-01-07T17:26:56.004712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Xoá đi các dữ liệu đã giải nén trên ổ cứng","metadata":{"papermill":{"duration":0.069897,"end_time":"2022-01-07T01:56:58.960972","exception":false,"start_time":"2022-01-07T01:56:58.891075","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!rm -r ./glove.840B.300d\n!rm -r ./GoogleNews-vectors-negative300\n!rm -r ./wiki-news-300d-1M\n!rm -r ./paragram_300_sl999","metadata":{"papermill":{"duration":3.853059,"end_time":"2022-01-07T01:57:02.884161","exception":false,"start_time":"2022-01-07T01:56:59.031102","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:27:02.355192Z","iopub.execute_input":"2022-01-07T17:27:02.355447Z","iopub.status.idle":"2022-01-07T17:27:06.243071Z","shell.execute_reply.started":"2022-01-07T17:27:02.355417Z","shell.execute_reply":"2022-01-07T17:27:06.242109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"embedding_matrix = np.mean([embedding_matrix_1, embedding_matrix_2, embedding_matrix_3], axis=0)\nnp.shape(embedding_matrix)","metadata":{"papermill":{"duration":1.024893,"end_time":"2022-01-07T01:57:03.986223","exception":false,"start_time":"2022-01-07T01:57:02.96133","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:27:06.245492Z","iopub.execute_input":"2022-01-07T17:27:06.246032Z","iopub.status.idle":"2022-01-07T17:27:07.150366Z","shell.execute_reply.started":"2022-01-07T17:27:06.245991Z","shell.execute_reply":"2022-01-07T17:27:07.149525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class MyDataset(Dataset):\n def __init__(self,dataset):\n self.dataset = dataset\n\n def __getitem__(self, index):\n data, target = self.dataset[index]\n\n return data, target, index\n def __len__(self):\n return len(self.dataset)","metadata":{"papermill":{"duration":0.079605,"end_time":"2022-01-07T01:57:04.135641","exception":false,"start_time":"2022-01-07T01:57:04.056036","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:27:07.151877Z","iopub.execute_input":"2022-01-07T17:27:07.152158Z","iopub.status.idle":"2022-01-07T17:27:07.157317Z","shell.execute_reply.started":"2022-01-07T17:27:07.152121Z","shell.execute_reply":"2022-01-07T17:27:07.156629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Mô hình\n\n","metadata":{"papermill":{"duration":0.069769,"end_time":"2022-01-07T01:57:04.275378","exception":false,"start_time":"2022-01-07T01:57:04.205609","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## 3.1 Cross Validation\nVấn đề : Bộ dataset chúng ta chỉ có 2 tập, train và test, không có tập val. Chúng ta cũng không được dùng tập train để kiểm thử mô hình, vì nó sẽ dẫn tới overfitting trên tập train. Vậy chúng ta sẽ lấy 1 phần của tập train ra làm tập validation. Nhưng, tập train của dữ liệu quá ít nhãn 1, việc lấy ra 1 phần của tập lỡ như hầu hết nhãn 1 đều nằm trong tập val này thì dữ liệu nhãn 1 ở tập train sẽ ít đi và dẫn đến thiếu dữ liệu train. Điều này dẫn đến mô hình không học được nhãn 1, có thể model không tốt . Để mô hình được huấn luyện tốt nhất, em đề xuất sử dụng Cross Validation.\n* Cross Validation là phương pháp chia nhỏ tập training ra thành N phần. Với mỗi lần train, mô hình sẽ sử dụng N-1 phần để train, sau đó test dựa trên 1 phần còn lại. Điều này sẽ giúp cho mô hình hạn chế gặp phải overfitting và giúp bạn tìm ra được những Hyper parameter tốt hơn.\n\n\n* Như hình chúng ta có thể thấy : Với mỗi lần train đầu, lấy 4 fold đầu tiên để train. Sau đó để val, sử dụng fold 5 để val. Qua lần train thứ 2, bạn lấy từ fold 2 đến fold 5 để train, rồi lại lấy fold 1 để val. Và đó, chính là Cross Validation.\n\n\n* Phương pháp đánh giá : Training data ta chia thành K phần. Sau đó train model K lần, mỗi lần train sẽ chọn 1 phần làm dữ liệu validation và K-1 phần còn lại làm dữ liệu training. Kết quả đánh giá model cuối cùng là trung bình cộng kết quả đánh giá của K lần train.\n\nEm chia dữ liệu thành 5 phần:","metadata":{"papermill":{"duration":0.069087,"end_time":"2022-01-07T01:57:04.415537","exception":false,"start_time":"2022-01-07T01:57:04.34645","status":"completed"},"tags":[]}},{"cell_type":"code","source":"splits = list(StratifiedKFold(n_splits=5, shuffle=True, random_state=2022).split(x_train, y_train))","metadata":{"papermill":{"duration":0.279802,"end_time":"2022-01-07T01:57:04.765093","exception":false,"start_time":"2022-01-07T01:57:04.485291","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:27:07.405645Z","iopub.execute_input":"2022-01-07T17:27:07.406185Z","iopub.status.idle":"2022-01-07T17:27:07.605613Z","shell.execute_reply.started":"2022-01-07T17:27:07.406147Z","shell.execute_reply":"2022-01-07T17:27:07.60488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2 ModelBiLSTM","metadata":{"papermill":{"duration":0.070003,"end_time":"2022-01-07T01:57:04.906206","exception":false,"start_time":"2022-01-07T01:57:04.836203","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Tham khảo : https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.htmlhttps://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html","metadata":{"papermill":{"duration":0.069523,"end_time":"2022-01-07T01:57:05.044901","exception":false,"start_time":"2022-01-07T01:57:04.975378","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### Mô hình\n\n* Mô hình gồm t input, các input được đưa vào mô hình đúng với thứ tự từ trong câu\n* Mỗi hình vuông được gọi là 1 state, đầu vào mỗi state là $ x_t $, $h_{t-1}$ với $h_t = f(W*x_t + U*h_{t-1})$. ($\\mathbf W$ là trọng số của đầu vào, $\\mathbf U$ là trọng số của trạng thái ẩn), $\\mathbb f$ là activation value như: sigmoid, tanh, ReLU,....\n* Có thể thấy $h_t$ mang cả thông tin từ hidden state trước \n* $h_0$ được thêm vào để cho chuẩn công thức nên thường được gán bằng 0 hoặc giá trị ngẫu nhiên\n* $y_t = g(V*h_t)$. V là trọng số của trạng thái ẩn sau khi tính đầu ra\n\n#### LSTM (Long short term memory)\n* Mạng RNN có yếu điểm là không mô tả học được chuỗi quá dài do hiện tượng triệt tiêu đạo hàm (vanishing gradient). Mạng LSTM ra đời khắc phục phần nào nhược điểm này bằng cách cho phép thông tin lan truyền trực tiếp hơn thông qua một biến trạng thái ô (cell state).\n\n* Mạng bộ nhớ dài-ngắn (Long Short Term Memory networks), thường được gọi là LSTM - là một dạng đặc biệt của RNN (Recurrent Neural Network), nó có khả năng học được các phụ thuộc xa. Chúng hoạt động cực kì hiệu quả trên nhiều bài toán khác nhau nên dần đã trở nên phổ biến như hiện nay.\n\n\n* LSTM được thiết kế để tránh được vấn đề phụ thuộc xa (long-term dependency). Việc nhớ thông tin trong suốt thời gian dài là đặc tính mặc định của chúng, chứ ta không cần phải huấn luyện nó để có thể nhớ được. Tức là ngay nội tại của nó đã có thể ghi nhớ được mà không cần bất kì can thiệp nào.\n\n\n\n\n\n* Tại state t\n * output: $c_t$ là cell state, $h_t$ là hidden state \n * input: $c_{t-1},h_{t-1}$. Ở đây $c$ là điểm mới so với RNN\n* Tính toán trong cell LSTM:\n * Cổng quên (forget gate): $\\mathbf f_t = \\sigma(\\mathbf W_{f} \\mathbf x_t + \\mathbf U_{f}\\mathbf h_{t-1})$\n * Cổng đầu vào (input gate): $\\mathbf i_t = \\sigma(\\mathbf W_{i} \\mathbf x_t + \\mathbf U_{i}\\mathbf h_{t-1})$\n * Cổng đầu ra (output gate): $\\mathbf o_t = \\sigma(\\mathbf W_{o} \\mathbf x_t + \\mathbf U_{o}\\mathbf h_{t-1})$\n * $\\tilde{\\mathbf c}_t = \\mathrm{tanh}(\\mathbf W_{c} \\mathbf x_t + \\mathbf U_{c}\\mathbf h_{t-1})$\n * Cổng trạng thái ô (cell state): $\\mathbf c_{t} = \\mathbf f_t \\times \\mathbf c_{t-1} + \\mathbf i_t \\times \\tilde{\\mathbf c}_t$. Forget gate quyết định xem lấy bao nhiêu từ cell state trước và input gate sẽ quyết định lấy bao nhiêu từ input của state và hidden state của state trước\n * $ \\mathbf h_t = \\mathrm{tanh}(c_t) \\times \\mathbf o_t $ , $\\mathbf y_t = \\phi_y(\\mathbf W_y \\mathbf h_t)$\n \n \n \n* Chìa khóa của LSTM là trạng thái tế bào (cell state) - chính đường chạy thông ngang phía trên của sơ đồ hình vẽ.Trạng thái tế bào là một dạng giống như băng truyền. Nó chạy xuyên suốt tất cả các mắt xích (các nút mạng) và chỉ tương tác tuyến tính đôi chút. Vì vậy mà các thông tin có thể dễ dàng truyền đi thông suốt mà không sợ bị thay đổi. $c_t$ sẽ được hiệu chỉnh để học được chuỗi dài hơn\n\n\n\n\n* \n\n\n* LSTM có khả năng bỏ đi hoặc thêm vào các thông tin cần thiết cho trạng thái tế báo, chúng được điều chỉnh cẩn thận bởi các nhóm được gọi là cổng (gate). Và đó là những khái niệm cốt lõi về LSTM.\n\n\n#### Bidirectional LSTM \n\n\n* Bidirectional LSTM là mô hình gồm hai LSTM: LSTM thứ nhất nhận đầu vào là chuỗi các từ theo thứ tự từ trái sang phải, LSTM còn lại nhận đầu vào là chuỗi các từ theo thứ tự từ phải sang trái. Cải thiện ngữ cảnh của mô hình, giúp mô hình học được tốt hơn\n\n\n\n#### Kiến trúc mô hình sử dụng\n* Lớp Embedding layer\n* BiLSTM\n* 2 lớp linear, với hàm kích hoạt relu\n* Dropout \n","metadata":{}},{"cell_type":"code","source":"hidden_size = 64\n\nclass BiLSTM(nn.Module):\n def __init__(self):\n super(BiLSTM, self).__init__()\n self.hidden_size = 64\n drp = 0.1\n self.embedding = nn.Embedding(vocab_size, embedding_dim)\n self.embedding.weight = nn.Parameter(torch.tensor(embedding_matrix, dtype=torch.float32))\n self.embedding.weight.requires_grad = False\n self.lstm = nn.LSTM(embedding_dim, self.hidden_size, bidirectional=True, batch_first=True)\n self.linear = nn.Linear(self.hidden_size*4 , 64)\n self.relu = nn.ReLU()\n self.dropout = nn.Dropout(drp)\n self.out = nn.Linear(64, tagset_size)\n def forward(self, x):\n #rint(x.size())*\n h_embedding = self.embedding(x)\n #_embedding = torch.squeeze(torch.unsqueeze(h_embedding, 0))*\n h_lstm, _ = self.lstm(h_embedding)\n avg_pool = torch.mean(h_lstm, 1)\n max_pool, _ = torch.max(h_lstm, 1)\n conc = torch.cat(( avg_pool, max_pool), 1)\n conc = self.relu(self.linear(conc))\n conc = self.dropout(conc)\n out = self.out(conc)\n return out","metadata":{"papermill":{"duration":0.083463,"end_time":"2022-01-07T01:57:05.197797","exception":false,"start_time":"2022-01-07T01:57:05.114334","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:27:10.369917Z","iopub.execute_input":"2022-01-07T17:27:10.370466Z","iopub.status.idle":"2022-01-07T17:27:10.38065Z","shell.execute_reply.started":"2022-01-07T17:27:10.370425Z","shell.execute_reply":"2022-01-07T17:27:10.379656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# matrix for the out-of-fold predictions\ntrain_preds = np.zeros((len(x_train)))\n# matrix for the predictions on the test set\ntest_preds = np.zeros((len(test_df)))\n\n# always call this before training for deterministic results\n# seed_everything()\n\nx_test_cuda = torch.tensor(x_test, dtype=torch.long).to(device)\ntest = torch.utils.data.TensorDataset(x_test_cuda)\ntest_loader = torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=False)\n\navg_losses_f = []\navg_val_losses_f = []\ntrain_loss = []\nvalid_loss = []\n\n# Tracking best validation accuracy\nbest_accuracy = 0","metadata":{"papermill":{"duration":4.558511,"end_time":"2022-01-07T01:57:09.826277","exception":false,"start_time":"2022-01-07T01:57:05.267766","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:39:55.1374Z","iopub.execute_input":"2022-01-07T17:39:55.137721Z","iopub.status.idle":"2022-01-07T17:39:55.320863Z","shell.execute_reply.started":"2022-01-07T17:39:55.137684Z","shell.execute_reply":"2022-01-07T17:39:55.320143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.3 Optimizers","metadata":{"papermill":{"duration":0.069785,"end_time":"2022-01-07T01:57:09.965785","exception":false,"start_time":"2022-01-07T01:57:09.896","status":"completed"},"tags":[]}},{"cell_type":"code","source":"base_lr, max_lr = 0.001, 0.003\ndef optimizers(model):\n optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=max_lr)\n return optimizer","metadata":{"papermill":{"duration":0.076948,"end_time":"2022-01-07T01:57:10.112674","exception":false,"start_time":"2022-01-07T01:57:10.035726","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:27:16.464379Z","iopub.execute_input":"2022-01-07T17:27:16.464623Z","iopub.status.idle":"2022-01-07T17:27:16.469312Z","shell.execute_reply.started":"2022-01-07T17:27:16.464579Z","shell.execute_reply":"2022-01-07T17:27:16.468661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.4 Loss Function","metadata":{"papermill":{"duration":0.07059,"end_time":"2022-01-07T01:57:10.2545","exception":false,"start_time":"2022-01-07T01:57:10.18391","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# define binary cross entropy loss\n# note that the model returns logit to take advantage of the log-sum-exp trick \n# for numerical stability in the loss\nloss_fn = torch.nn.BCEWithLogitsLoss(reduction='sum')","metadata":{"papermill":{"duration":0.077814,"end_time":"2022-01-07T01:57:10.402648","exception":false,"start_time":"2022-01-07T01:57:10.324834","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:27:16.470604Z","iopub.execute_input":"2022-01-07T17:27:16.471091Z","iopub.status.idle":"2022-01-07T17:27:16.479803Z","shell.execute_reply.started":"2022-01-07T17:27:16.471055Z","shell.execute_reply":"2022-01-07T17:27:16.479067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Training","metadata":{"papermill":{"duration":0.070498,"end_time":"2022-01-07T01:57:10.544589","exception":false,"start_time":"2022-01-07T01:57:10.474091","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## 4.1 Create PyTorch DataLoader","metadata":{}},{"cell_type":"code","source":"def create_data_loader(x_train,y_train,train_idx, valid_idx,batch_size):\n # split data in train / validation according to the KFold indeces\n # also, convert them to a torch tensor and store them on the GPU (done with .cuda())\n x_train = np.array(x_train)\n y_train = np.array(y_train)\n \n \n x_train_fold = torch.tensor(x_train[train_idx.astype(int)], dtype=torch.long).to(device)\n y_train_fold = torch.tensor(y_train[train_idx.astype(int), np.newaxis], dtype=torch.float32).to(device)\n \n x_val_fold = torch.tensor(x_train[valid_idx.astype(int)], dtype=torch.long).to(device)\n y_val_fold = torch.tensor(y_train[valid_idx.astype(int), np.newaxis], dtype=torch.float32).to(device)\n \n #############################################################################################\n train = torch.utils.data.TensorDataset(x_train_fold, y_train_fold)\n valid = torch.utils.data.TensorDataset(x_val_fold, y_val_fold)\n \n train = MyDataset(train)\n valid = MyDataset(valid)\n \n ##No need to shuffle the data again here. Shuffling happens when splitting for kfolds.\n train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=True)\n valid_loader = torch.utils.data.DataLoader(valid, batch_size=batch_size, shuffle=False)\n return train_loader,valid_loader","metadata":{"execution":{"iopub.status.busy":"2022-01-07T17:27:18.024175Z","iopub.execute_input":"2022-01-07T17:27:18.024428Z","iopub.status.idle":"2022-01-07T17:27:18.035879Z","shell.execute_reply.started":"2022-01-07T17:27:18.024401Z","shell.execute_reply":"2022-01-07T17:27:18.035158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2 Evaluate model","metadata":{}},{"cell_type":"code","source":"#eval model\ndef evaluate(model, valid_loader):\n \n # set evaluation mode of the model. This disabled operations which are only applied during training like dropout\n model.eval()\n # predict all the samples in y_val_fold batch per batch\n valid_preds_fold = np.zeros(len(valid_idx))\n avg_val_loss = 0\n val_accuracy = []\n \n for i, (x_batch, y_batch, index) in enumerate(valid_loader):\n \n with torch.no_grad():\n y_pred = model(x_batch).detach()\n \n avg_val_loss += loss_fn(y_pred, y_batch).item() / len(valid_loader)\n valid_preds_fold[i * batch_size:(i+1) * batch_size] = torch.special.expit(y_pred).cpu().numpy()[:, 0]\n if i==0 and epoch == 0:\n valid_loss.append(loss_fn(y_pred, y_batch))\n train_preds[valid_idx] = valid_preds_fold\n # Get the predictions\n preds = torch.argmax(y_pred, dim=1).flatten()\n\n # Calculate the accuracy rate\n accuracy = (preds == y_batch).cpu().numpy().mean() * 100\n val_accuracy.append(accuracy)\n accuracy = sum(val_accuracy)/len(val_accuracy)\n \n print('val_accuracy = {:.4f} '.format(accuracy))\n return valid_preds_fold, avg_val_loss\n#predicts model\ndef predict(model, test_loader):\n # predict all samples in the test set batch per batch\n test_preds_fold = np.zeros((len(test_df)))\n for i, (x_batch,) in enumerate(test_loader):\n \n y_pred = model(x_batch).detach()\n test_preds_fold[i * batch_size:(i+1) * batch_size] = torch.special.expit(y_pred).cpu().numpy()[:, 0]\n return test_preds_fold","metadata":{"papermill":{"duration":0.086219,"end_time":"2022-01-07T01:57:10.701934","exception":false,"start_time":"2022-01-07T01:57:10.615715","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:39:41.668682Z","iopub.execute_input":"2022-01-07T17:39:41.668957Z","iopub.status.idle":"2022-01-07T17:39:41.680405Z","shell.execute_reply.started":"2022-01-07T17:39:41.668924Z","shell.execute_reply":"2022-01-07T17:39:41.679266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2 Training","metadata":{}},{"cell_type":"code","source":"n_epochs = 5\nset_seed()\nfor i, (train_idx, valid_idx) in enumerate(splits):\n train_loader,valid_loader = create_data_loader(x_train,y_train,train_idx, valid_idx,batch_size)\n model = BiLSTM()\n\n # make sure everything in the model is running on the GPU\n model.to(device)\n \n optimizer = optimizers(model)\n print(f'Fold {i + 1}')\n for epoch in range(n_epochs):\n # =======================================\n # Training\n # =======================================\n start_time = time.time()\n # set train mode of the model. This enables operations which are only applied during training like dropout\n \n model.train()\n avg_loss = 0. \n for i, (x_batch, y_batch, index) in enumerate(train_loader):\n \n # Forward pass: compute predicted y by passing x to the model. \n y_pred = model(x_batch)\n \n # Compute and print loss.\n loss = loss_fn(y_pred, y_batch)\n\n # Before the backward pass, use the optimizer object to zero all of the\n # gradients for the Tensors it will update (which are the learnable weights\n # of the model)\n optimizer.zero_grad()\n # Backward pass: compute gradient of the loss with respect to model parameters\n loss.backward()\n\n # Calling the step function on an Optimizer makes an update to its parameters\n optimizer.step()\n avg_loss += loss.item()\n if(epoch==0 and i==0):\n train_loss.append(loss.item())\n \n # Calculate the average loss over the entire training data \n avg_loss = avg_loss / len(train_loader) \n \n \n # =======================================\n # Evaluation\n # =======================================\n if valid_loader is not None:\n # After the completion of each training epoch, measure the model's\n # performance on our validation set.\n valid_preds_fold, avg_val_loss = evaluate(model, valid_loader)\n elapsed_time = time.time() - start_time \n print('Epoch {}/{} \\t avg_loss={:.4f} \\t avg_val_loss={:.4f} \\t time={:.2f}s'.format(\n epoch + 1, n_epochs, avg_loss, avg_val_loss, elapsed_time))\n \n \n avg_losses_f.append(avg_loss)\n avg_val_losses_f.append(avg_val_loss) \n #predics test_data_loader\n test_preds_fold = predict(model, test_loader)\n test_preds += test_preds_fold / len(splits)\n \n ##predics train_data_loader\n train_preds[valid_idx] = valid_preds_fold\n\nprint('All \\t loss={:.4f} \\t val_loss={:.4f} \\t '.format(np.average(avg_losses_f),np.average(avg_val_losses_f)))","metadata":{"execution":{"iopub.status.busy":"2022-01-07T17:43:54.740445Z","iopub.execute_input":"2022-01-07T17:43:54.74072Z","iopub.status.idle":"2022-01-07T17:54:31.79345Z","shell.execute_reply.started":"2022-01-07T17:43:54.740681Z","shell.execute_reply":"2022-01-07T17:54:31.792514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots()\navg_losses_f_n=np.array(avg_losses_f)\navg_val_losses_f_n=np.array(avg_val_losses_f)\nplt.plot(avg_losses_f_n, label='avg_losses_f')\nplt.plot(avg_val_losses_f_n, label='avg_val_losses_f')\nplt.title(\"Avg Losses\")\nplt.legend()","metadata":{"papermill":{"duration":0.490555,"end_time":"2022-01-07T02:10:59.477312","exception":false,"start_time":"2022-01-07T02:10:58.986757","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:54:35.650737Z","iopub.execute_input":"2022-01-07T17:54:35.651446Z","iopub.status.idle":"2022-01-07T17:54:35.872325Z","shell.execute_reply.started":"2022-01-07T17:54:35.651408Z","shell.execute_reply":"2022-01-07T17:54:35.871643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots()\ntrain_loss = np.array(train_loss)\nvalid_loss = np.array(valid_loss)\nplt.plot(train_loss, label='train_loss ford 1')\nplt.plot(valid_loss, label='valid_loss ford 1')\nplt.title(\"Training Losses\")\nplt.legend()","metadata":{"papermill":{"duration":0.38916,"end_time":"2022-01-07T02:10:59.999859","exception":false,"start_time":"2022-01-07T02:10:59.610699","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:54:38.210695Z","iopub.execute_input":"2022-01-07T17:54:38.211453Z","iopub.status.idle":"2022-01-07T17:54:38.439999Z","shell.execute_reply.started":"2022-01-07T17:54:38.211412Z","shell.execute_reply":"2022-01-07T17:54:38.439294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def bestThresshold(y_train,train_preds):\n tmp = [0,0,0] # idx, cur, max\n delta = 0\n for tmp[0] in tqdm(np.arange(0.1, 0.501, 0.01)):\n tmp[1] = f1_score(y_train, np.array(train_preds)>tmp[0])\n if tmp[1] > tmp[2]:\n delta = tmp[0]\n tmp[2] = tmp[1]\n print('best threshold is {:.4f} with F1 score: {:.4f}'.format(delta, tmp[2]))\n return delta\ndelta = bestThresshold(y_train,train_preds)","metadata":{"papermill":{"duration":16.486034,"end_time":"2022-01-07T02:11:16.564203","exception":false,"start_time":"2022-01-07T02:11:00.078169","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:54:40.621513Z","iopub.execute_input":"2022-01-07T17:54:40.622339Z","iopub.status.idle":"2022-01-07T17:54:57.273545Z","shell.execute_reply.started":"2022-01-07T17:54:40.622294Z","shell.execute_reply":"2022-01-07T17:54:57.272812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_test = y_train\ntrain_preds\ntrain_preds_test = (train_preds > delta).astype(int)\n\nunique, counts = np.unique(train_preds_test, return_counts=True)\ndict(zip(unique, counts))\n# np.amax(train_preds_test)\n","metadata":{"papermill":{"duration":0.108672,"end_time":"2022-01-07T02:11:16.751755","exception":false,"start_time":"2022-01-07T02:11:16.643083","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:56:04.483959Z","iopub.execute_input":"2022-01-07T17:56:04.484487Z","iopub.status.idle":"2022-01-07T17:56:04.516936Z","shell.execute_reply.started":"2022-01-07T17:56:04.484444Z","shell.execute_reply":"2022-01-07T17:56:04.515969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(classification_report(y_train_test, train_preds_test))","metadata":{"papermill":{"duration":1.682665,"end_time":"2022-01-07T02:11:18.513426","exception":false,"start_time":"2022-01-07T02:11:16.830761","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:56:06.337205Z","iopub.execute_input":"2022-01-07T17:56:06.337475Z","iopub.status.idle":"2022-01-07T17:56:07.871421Z","shell.execute_reply.started":"2022-01-07T17:56:06.337449Z","shell.execute_reply":"2022-01-07T17:56:07.870538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(confusion_matrix(y_train_test, train_preds_test))","metadata":{"papermill":{"duration":2.033378,"end_time":"2022-01-07T02:11:20.642245","exception":false,"start_time":"2022-01-07T02:11:18.608867","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:56:10.345999Z","iopub.execute_input":"2022-01-07T17:56:10.346764Z","iopub.status.idle":"2022-01-07T17:56:11.967707Z","shell.execute_reply.started":"2022-01-07T17:56:10.346711Z","shell.execute_reply":"2022-01-07T17:56:11.966909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = test_df[['qid']].copy()\nsubmission['prediction'] = (test_preds > delta).astype(int)\nsubmission.to_csv('submission.csv', index=False)","metadata":{"papermill":{"duration":0.944539,"end_time":"2022-01-07T02:11:21.676549","exception":false,"start_time":"2022-01-07T02:11:20.73201","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-07T17:56:12.032301Z","iopub.execute_input":"2022-01-07T17:56:12.032522Z","iopub.status.idle":"2022-01-07T17:56:12.836606Z","shell.execute_reply.started":"2022-01-07T17:56:12.032495Z","shell.execute_reply":"2022-01-07T17:56:12.83587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!head submission.csv","metadata":{"execution":{"iopub.execute_input":"2022-01-07T02:11:21.842777Z","iopub.status.busy":"2022-01-07T02:11:21.841976Z","iopub.status.idle":"2022-01-07T02:11:22.633334Z","shell.execute_reply":"2022-01-07T02:11:22.632571Z","shell.execute_reply.started":"2022-01-06T18:47:54.065748Z"},"papermill":{"duration":0.876328,"end_time":"2022-01-07T02:11:22.633469","exception":false,"start_time":"2022-01-07T02:11:21.757141","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]}]}