{"cells":[{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":false,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.execute_input":"2020-10-13T17:24:00.28949Z","iopub.status.busy":"2020-10-13T17:24:00.288771Z","iopub.status.idle":"2020-10-13T17:24:01.413171Z","shell.execute_reply":"2020-10-13T17:24:01.412399Z"},"papermill":{"duration":1.149168,"end_time":"2020-10-13T17:24:01.4133","exception":false,"start_time":"2020-10-13T17:24:00.264132","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import fasttext, string, collections\nimport pandas as pd, numpy as np\nfrom sklearn.manifold import TSNE\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.017606,"end_time":"2020-10-13T17:24:01.451299","exception":false,"start_time":"2020-10-13T17:24:01.433693","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Embeddings for Knowledge Tracing\n\n**Motivation comes from [this paper](https://arxiv.org/pdf/2005.12442.pdf). I want to treat the questions as words in a corpus, so that I can embed them such that similar questions are clustered together in the same way that we use GloVe or Word2Vec vectors to cluster words with similar meaning together.** \n\n**Facebook's [fastText](https://github.com/facebookresearch/fastText) makes use of character level information. Each word is represented as a bag-of-character *n*-gram as well as the word itself. For example, the word `queen` with *n*-gram=3 is composed of `<qu`, `que`, `uee`, `een`, and `en>`. This can allow you to capture the meaning of suffixes/prefixes, which motivates the encoding procedure I carry out in this notebook.**\n\n**fastText can also generate embeddings for words/questions not in our corpus by adding the character n-gram of all n-gram representations. It essentially builds a representation for OOV tokens by stitching together character level n-grams that it has seen during training.**"},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:24:01.495851Z","iopub.status.busy":"2020-10-13T17:24:01.495078Z","iopub.status.idle":"2020-10-13T17:24:21.023876Z","shell.execute_reply":"2020-10-13T17:24:21.023215Z"},"papermill":{"duration":19.554752,"end_time":"2020-10-13T17:24:21.024003","exception":false,"start_time":"2020-10-13T17:24:01.469251","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#full length of dataset is 101,230,332\nN_SAMPLES = 1000000\n\ntrain = pd.read_pickle('../input/riiid-answer-correctness-prediction-files/train.pkl')\ntrain = train[train['content_type_id'] == 0]\n\nquestions = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv').rename({'question_id':'content_id'},\n                                                                                        axis=1)\n\nprint(train.shape)\ntrain = train.sample(N_SAMPLES, random_state=34)\nprint(train.shape)\ntrain = pd.merge(train, questions, how='left', on='content_id')\nprint(train.shape)\nprint(train['user_id'].nunique())\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.02072,"end_time":"2020-10-13T17:24:21.065489","exception":false,"start_time":"2020-10-13T17:24:21.044769","status":"completed"},"tags":[]},"cell_type":"markdown","source":"**As per the [data description tab](https://www.kaggle.com/c/riiid-test-answer-prediction/data), the `tags` of the questions found in the `questions.csv` of the competition are sufficient for clustering the questions together. In total, we have the following features to play around with to group questions together. We can of course engineer more clustering features, but for now we stick with these:**\n\n**Train**\n* `content_id` - (int16) ID code for the user interaction\n\n**Questions**\n* `tags` - one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together.\n* `bundle_id` - code for which questions are served together.\n* `part` - the relevant section of the TOEIC test\n\n**Let's quickly explore these features. We begin with `content_id`. Bare in mind I am using a specific subsample of `train.csv` defined by the first block of the notebook.**"},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:24:21.112538Z","iopub.status.busy":"2020-10-13T17:24:21.111707Z","iopub.status.idle":"2020-10-13T17:24:21.126581Z","shell.execute_reply":"2020-10-13T17:24:21.125817Z"},"papermill":{"duration":0.040426,"end_time":"2020-10-13T17:24:21.126715","exception":false,"start_time":"2020-10-13T17:24:21.086289","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"print(f\"There are {train['content_id'].nunique()} unique content ids in train.csv\")\nprint(f\"On average, students get {round(train['answered_correctly'].mean(), 3)} of questions correct\")","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:24:21.17483Z","iopub.status.busy":"2020-10-13T17:24:21.174085Z","iopub.status.idle":"2020-10-13T17:24:21.197883Z","shell.execute_reply":"2020-10-13T17:24:21.197175Z"},"papermill":{"duration":0.050405,"end_time":"2020-10-13T17:24:21.198036","exception":false,"start_time":"2020-10-13T17:24:21.147631","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"train['content_id'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.020571,"end_time":"2020-10-13T17:24:21.240109","exception":false,"start_time":"2020-10-13T17:24:21.219538","status":"completed"},"tags":[]},"cell_type":"markdown","source":"**Now we move onto the `tags` feature. Since there are multiple tags per questions, I mainly want to know what the maximum number of tags a given question can have and how many unique tags there are.**"},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:24:21.312547Z","iopub.status.busy":"2020-10-13T17:24:21.307302Z","iopub.status.idle":"2020-10-13T17:24:21.71548Z","shell.execute_reply":"2020-10-13T17:24:21.714897Z"},"papermill":{"duration":0.453606,"end_time":"2020-10-13T17:24:21.715607","exception":false,"start_time":"2020-10-13T17:24:21.262001","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"max_ = 0\niter_ = 0\nfor i, row in enumerate(train['tags'].values):\n    if len(row.split()) > max_: \n        max_ = len(row.split())\n        iter_ = i\ntrain.iloc[iter_]","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:24:21.764113Z","iopub.status.busy":"2020-10-13T17:24:21.76349Z","iopub.status.idle":"2020-10-13T17:26:50.983447Z","shell.execute_reply":"2020-10-13T17:26:50.98235Z"},"papermill":{"duration":149.247167,"end_time":"2020-10-13T17:26:50.983618","exception":false,"start_time":"2020-10-13T17:24:21.736451","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"tags = []\n\nfor row in tqdm(train.index):\n    tags.append(train.iloc[row]['tags'].split())","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:26:51.889508Z","iopub.status.busy":"2020-10-13T17:26:51.863565Z","iopub.status.idle":"2020-10-13T17:26:52.176797Z","shell.execute_reply":"2020-10-13T17:26:52.177262Z"},"papermill":{"duration":0.763875,"end_time":"2020-10-13T17:26:52.17744","exception":false,"start_time":"2020-10-13T17:26:51.413565","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#check nunique tags\nlen(set([tag for sublist in tags for tag in sublist]))","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.427481,"end_time":"2020-10-13T17:26:53.036561","exception":false,"start_time":"2020-10-13T17:26:52.60908","status":"completed"},"tags":[]},"cell_type":"markdown","source":"**Let's also look at the distribution of `part` and `bundle_id`:**"},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:26:53.951308Z","iopub.status.busy":"2020-10-13T17:26:53.950601Z","iopub.status.idle":"2020-10-13T17:26:53.955366Z","shell.execute_reply":"2020-10-13T17:26:53.954871Z"},"papermill":{"duration":0.492865,"end_time":"2020-10-13T17:26:53.955489","exception":false,"start_time":"2020-10-13T17:26:53.462624","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"questions['part'].value_counts(normalize=True)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:26:54.823935Z","iopub.status.busy":"2020-10-13T17:26:54.823116Z","iopub.status.idle":"2020-10-13T17:26:54.841807Z","shell.execute_reply":"2020-10-13T17:26:54.842353Z"},"papermill":{"duration":0.463695,"end_time":"2020-10-13T17:26:54.842523","exception":false,"start_time":"2020-10-13T17:26:54.378828","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"print(questions['bundle_id'].nunique())\nquestions['bundle_id'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.443297,"end_time":"2020-10-13T17:26:55.729275","exception":false,"start_time":"2020-10-13T17:26:55.285978","status":"completed"},"tags":[]},"cell_type":"markdown","source":"**Okay, maybe too many unique `bundle_ids` to use for clustering, but we can definitely use `tags` and potentially `part`, although the it is dominated by the value `5`.**\n\n**Now, it is important for the embedding strategy I wish to test that each unique question tag is mapped to a single character. You will see why shortly. If you run `string.printable`, it will only give you 100 characters to use, so we are going to need some other characters. Let's go to the Greek alphabet first and maybe some Cyrillic/Latin letters if we need them. If anyone has a cleaner way of generating characters, do tell.**"},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-13T17:26:56.585864Z","iopub.status.busy":"2020-10-13T17:26:56.585067Z","iopub.status.idle":"2020-10-13T17:26:56.588929Z","shell.execute_reply":"2020-10-13T17:26:56.58949Z"},"papermill":{"duration":0.434076,"end_time":"2020-10-13T17:26:56.58965","exception":false,"start_time":"2020-10-13T17:26:56.155574","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"greek = '\\u03A9\\u0394\\u03BC\\u00B0\\u0302\\u03C0\\u03F4\\u03BB\\u03B1\\u03B3\\u03B4\\u03B5\\u03B6\\u03B7\\u03B8\\u03B9\\u03BA\\u03BD\\u03BE\\u03C1\\u03C2\\u03C3\\u03C4\\u03C5\\u03C6\\u03C7\\u03C8\\u03C9\\u0391\\u0395\\u0396\\u0397\\u0398\\u0399\\u039A\\u039B\\u039C\\u039D\\u039E\\u039F\\u03A0\\u03A1\\u03A3\\u03A4\\u03A5\\u03A6\\u03A7\\u03A8'\ngreek","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-13T17:26:57.435898Z","iopub.status.busy":"2020-10-13T17:26:57.435051Z","iopub.status.idle":"2020-10-13T17:26:57.439563Z","shell.execute_reply":"2020-10-13T17:26:57.438922Z"},"papermill":{"duration":0.42982,"end_time":"2020-10-13T17:26:57.439679","exception":false,"start_time":"2020-10-13T17:26:57.009859","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"cyrillic = '\\u0410\\u0430\\u0411\\u0413\\u0414\\u0415\\u0416\\u0417\\u0418\\u0419\\u0431\\u0432\\u0433\\u0434\\u0435\\u0436\\u0437\\u0438\\u0439\\u041a\\u041b\\u041c\\u041d\\u041e\\u041f\\u043a\\u043b\\u043c\\u043d\\u043e\\u043f\\u0420\\u0421\\u0422\\u0423\\u0424\\u0425\\u0426\\u0427\\u0428\\u0429'\ncyrillic","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-13T17:26:58.289997Z","iopub.status.busy":"2020-10-13T17:26:58.289133Z","iopub.status.idle":"2020-10-13T17:26:58.293923Z","shell.execute_reply":"2020-10-13T17:26:58.293181Z"},"papermill":{"duration":0.434153,"end_time":"2020-10-13T17:26:58.294044","exception":false,"start_time":"2020-10-13T17:26:57.859891","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"latin = '\\u00A1\\u00A2\\u00A3\\u00A4\\u00A5\\u00A6\\u00A7\\u00A8\\u00A9\\u00B0\\u00B1\\u00B2\\u00B3\\u00B4\\u00B5\\u00B6\\u00B7\\u00B8\\u00B9\\u00C0\\u00C1\\u00C3\\u00C5\\u00C6\\u00C7\\u00C8\\u00C9\\u00D0\\u00D1\\u00D2\\u00D4\\u00D5\\u00D6\\u00D7\\u00D8\\u00D9'\nlatin","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:26:59.145992Z","iopub.status.busy":"2020-10-13T17:26:59.145236Z","iopub.status.idle":"2020-10-13T17:26:59.149656Z","shell.execute_reply":"2020-10-13T17:26:59.149045Z"},"papermill":{"duration":0.434281,"end_time":"2020-10-13T17:26:59.149775","exception":false,"start_time":"2020-10-13T17:26:58.715494","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#not using 012 because they are reserved for labels\nchars = '3456789' + string.ascii_letters + greek + cyrillic + latin\nlen(chars)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:27:00.035561Z","iopub.status.busy":"2020-10-13T17:27:00.014764Z","iopub.status.idle":"2020-10-13T17:27:00.782351Z","shell.execute_reply":"2020-10-13T17:27:00.781694Z"},"papermill":{"duration":1.209913,"end_time":"2020-10-13T17:27:00.782474","exception":false,"start_time":"2020-10-13T17:26:59.572561","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"most_common = [word for word, word_count in collections.Counter([tag for sublist in tags for tag in sublist]).most_common(len(chars))]\nleast_common = [word for word, word_count in collections.Counter([tag for sublist in tags for tag in sublist]).most_common()[len(chars):]]\nlen(most_common)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:27:01.679886Z","iopub.status.busy":"2020-10-13T17:27:01.679227Z","iopub.status.idle":"2020-10-13T17:27:01.682421Z","shell.execute_reply":"2020-10-13T17:27:01.681793Z"},"papermill":{"duration":0.433914,"end_time":"2020-10-13T17:27:01.682538","exception":false,"start_time":"2020-10-13T17:27:01.248624","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"char_dict = {}\n\nfor tag, char in zip(most_common, chars):\n    char_dict[tag] = char","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:27:02.543002Z","iopub.status.busy":"2020-10-13T17:27:02.542253Z","iopub.status.idle":"2020-10-13T17:27:03.933862Z","shell.execute_reply":"2020-10-13T17:27:03.933162Z"},"papermill":{"duration":1.822026,"end_time":"2020-10-13T17:27:03.933985","exception":false,"start_time":"2020-10-13T17:27:02.111959","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#remove low frequency tags and maps to unique character\ndef filter_tags(tag):\n    return_tag = []\n    \n    for tag in tag.split():\n        if tag not in least_common:\n            tag = char_dict.get(tag)\n            return_tag.append(tag)\n            \n    return \" \".join(return_tag)\n     \ntrain['tags'] = train['tags'].apply(filter_tags)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.423333,"end_time":"2020-10-13T17:27:04.784973","exception":false,"start_time":"2020-10-13T17:27:04.36164","status":"completed"},"tags":[]},"cell_type":"markdown","source":"**After mapping each tag to one of the characters in `string.printable`, we then 'pad' our tags such that even those with only 1 tag can be compared to those with 6. Now we can string concat the answer of the question so that the model learns it as a sort of word suffix.**"},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:27:05.693023Z","iopub.status.busy":"2020-10-13T17:27:05.688733Z","iopub.status.idle":"2020-10-13T17:27:07.309409Z","shell.execute_reply":"2020-10-13T17:27:07.30881Z"},"papermill":{"duration":2.096625,"end_time":"2020-10-13T17:27:07.309549","exception":false,"start_time":"2020-10-13T17:27:05.212924","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#I know I know, this is sloppy\ndef pad_tags(tag):\n    if len(tag.split()) == 1:\n        return \"\".join([tag]*6)\n    \n    if len(tag.split()) == 2:\n        tag1 = tag.split()[0]\n        tag2 = tag.split()[1]\n        return \"\".join([tag1, tag2]*3)\n    \n    if len(tag.split()) == 3:\n        tag1 = tag.split()[0]\n        tag2 = tag.split()[1]\n        tag3 = tag.split()[2]\n        return \"\".join([tag1, tag2, tag3]*2)\n    \n    if len(tag.split()) == 4:\n        tag1 = tag.split()[0]\n        tag2 = tag.split()[1]\n        tag3 = tag.split()[2]\n        tag4 = tag.split()[3]\n        return \"\".join([tag1, tag2, tag3, tag4, tag1, tag2])\n    \n    if len(tag.split()) == 5:\n        tag1 = tag.split()[0]\n        tag2 = tag.split()[1]\n        tag3 = tag.split()[2]\n        tag4 = tag.split()[3]\n        tag5 = tag.split()[4]\n        return \"\".join([tag1, tag2, tag3, tag4, tag5, tag1])\n    \n    else: return tag      \n    \ntrain['tags'] = train['tags'].apply(pad_tags)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.434285,"end_time":"2020-10-13T17:27:08.180595","exception":false,"start_time":"2020-10-13T17:27:07.74631","status":"completed"},"tags":[]},"cell_type":"markdown","source":"**I will also changed the `answered_correctly = -1` value to `2`. I want to include this is a category on par with the other answers, so that we can cluster questions together that students did not answer. This makes sense if there is some logic behind the lack of an answer: was the question too hard or poorly worded? I do not know if any such logic underlies these `-1` values, but I will assume (for now) there is.**"},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:27:09.07957Z","iopub.status.busy":"2020-10-13T17:27:09.078381Z","iopub.status.idle":"2020-10-13T17:27:09.732123Z","shell.execute_reply":"2020-10-13T17:27:09.731583Z"},"papermill":{"duration":1.111394,"end_time":"2020-10-13T17:27:09.732263","exception":false,"start_time":"2020-10-13T17:27:08.620869","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"train['answered_correctly'] = train['answered_correctly'].apply(str)\ntrain['answered_correctly'] = train['answered_correctly'].replace({'-1':'2'})\ntrain['interaction_enc'] = train['tags'] + train['answered_correctly']","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:27:10.616288Z","iopub.status.busy":"2020-10-13T17:27:10.615543Z","iopub.status.idle":"2020-10-13T17:27:10.620545Z","shell.execute_reply":"2020-10-13T17:27:10.620005Z"},"papermill":{"duration":0.452103,"end_time":"2020-10-13T17:27:10.62067","exception":false,"start_time":"2020-10-13T17:27:10.168567","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#debugging step\ni = 0\nfor i in range(100000):\n    if len([_ for _ in train.iloc[i]['interaction_enc']]) != 7: print('stop'); print(i); break","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:27:11.498496Z","iopub.status.busy":"2020-10-13T17:27:11.49746Z","iopub.status.idle":"2020-10-13T17:33:18.249906Z","shell.execute_reply":"2020-10-13T17:33:18.248004Z"},"papermill":{"duration":367.190389,"end_time":"2020-10-13T17:33:18.250062","exception":false,"start_time":"2020-10-13T17:27:11.059673","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"with open('corpus.txt', 'w') as file:\n    for user in tqdm(train['user_id'].unique()):\n        user_df = train[train['user_id'] == user]\n        line=' '.join(user_df['interaction_enc'].values)\n        file.write(line+'\\n')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":1.468368,"end_time":"2020-10-13T17:33:21.219087","exception":false,"start_time":"2020-10-13T17:33:19.750719","status":"completed"},"tags":[]},"cell_type":"markdown","source":"**We can train a skipgram or a continuous bag-of-words (CBOW) model. In general, CBOW learns better syntactic relationships between words while skipgram is better at capturing semantic relationships. Suppose we have the word `general`. CBOW would fetch morphologically similar words, like `generalize` or `generalized`, whereas the skipgram would find words like `universal` and `common`. As we don't really have words here, let's just try both.**\n\n**From the [fastText documentation](https://fasttext.cc/docs/en/python-module.html), we can play with the following parameters. Values in `[]` are default values.**\n\n* **input** - training file path (required)\n* **model** - unsupervised fasttext model {cbow, skipgram} [skipgram]\n* **lr** - learning rate [0.05]\n* **dim** - size of word vectors [100]\n* **ws** - size of the context window [5]\n* **epoch** - number of epochs [5]\n* **minCount** - minimal number of word occurences [5]\n* **minn** - min length of char ngram [3]\n* **maxn** - max length of char ngram [6]\n* **neg** - number of negatives sampled [5]\n* **wordNgrams** - max length of word ngram [1]\n* **loss** - loss function {ns, hs, softmax, ova} [ns]\n* **bucket** - number of buckets [2000000]\n* **thread** - number of threads [number of cpus]\n* **lrUpdateRate** - change the rate of updates for the learning rate [100]\n* **t** - sampling threshold [0.0001]\n* **verbose** - verbose [2]\n    \n**This is why I formatted the encoded question/answer pairs the way I did: we can just set `minn=1` and `maxn=1` to capture similarities between the tags associated to a particular problem and how the user answered it.** "},{"metadata":{"trusted":true},"cell_type":"code","source":"DIM = 200\nWINDOW = 6\nPERPLEXITY = 40\nN_ITER = 2500\nSEED = 34","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:33:24.161633Z","iopub.status.busy":"2020-10-13T17:33:24.160777Z","iopub.status.idle":"2020-10-13T17:34:00.711946Z","shell.execute_reply":"2020-10-13T17:34:00.712547Z"},"papermill":{"duration":38.005723,"end_time":"2020-10-13T17:34:00.71272","exception":false,"start_time":"2020-10-13T17:33:22.706997","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"%%time\n\ncbow = fasttext.train_unsupervised('corpus.txt', model='cbow',\n                                    dim=DIM, minn=1, maxn=1, ws=WINDOW)\nskipgram = fasttext.train_unsupervised('corpus.txt', model='skipgram',\n                                    dim=DIM, minn=1, maxn=1, ws=WINDOW)\nprint(f\"{cbow.get_output_matrix().mean()}\")                                                                                                                    \nprint(f\"{skipgram.get_output_matrix().mean()}\")","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:34:03.528516Z","iopub.status.busy":"2020-10-13T17:34:03.527511Z","iopub.status.idle":"2020-10-13T17:34:03.531569Z","shell.execute_reply":"2020-10-13T17:34:03.530943Z"},"papermill":{"duration":1.408855,"end_time":"2020-10-13T17:34:03.531684","exception":false,"start_time":"2020-10-13T17:34:02.122829","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#sanity check\ncbow.get_subwords('Xy5Xy51')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":1.406561,"end_time":"2020-10-13T17:34:06.332017","exception":false,"start_time":"2020-10-13T17:34:04.925456","status":"completed"},"tags":[]},"cell_type":"markdown","source":"**Great, seems that the CBOW model is forming character *n*-grams for each unique tag ID and question answer just as we wanted. Now let's visualize these embeddings with t-SNE to see what sort of structure there is.**"},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-13T17:34:09.265744Z","iopub.status.busy":"2020-10-13T17:34:09.265027Z","iopub.status.idle":"2020-10-13T17:34:09.269012Z","shell.execute_reply":"2020-10-13T17:34:09.268289Z"},"papermill":{"duration":1.457048,"end_time":"2020-10-13T17:34:09.26915","exception":false,"start_time":"2020-10-13T17:34:07.812102","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/jeffd23/visualizing-word-vectors-with-t-sne/comments\ndef tsne_plot(model):\n    labels = []\n    tokens = []\n\n    for word in model.words:\n        tokens.append(model[word])\n        labels.append(word)\n    \n    tsne_model = TSNE(perplexity=PERPLEXITY, n_components=2, init='pca', \n                      n_iter=N_ITER, n_jobs=-1, random_state=SEED)\n    new_values = tsne_model.fit_transform(tokens)\n\n    x = []\n    y = []\n    for value in new_values:\n        x.append(value[0])\n        y.append(value[1])\n        \n    plt.figure(figsize=(15, 15)) \n    for i in range(len(x)):\n        plt.scatter(x[i],y[i])\n        \n        if i%50==0:\n            plt.annotate(labels[i],\n                         xy=(x[i], y[i]),\n                         xytext=(5, 2),\n                         textcoords='offset points',\n                         ha='right',\n                         va='bottom')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:34:12.283398Z","iopub.status.busy":"2020-10-13T17:34:12.282704Z","iopub.status.idle":"2020-10-13T17:34:59.447229Z","shell.execute_reply":"2020-10-13T17:34:59.447801Z"},"papermill":{"duration":48.64902,"end_time":"2020-10-13T17:34:59.447955","exception":false,"start_time":"2020-10-13T17:34:10.798935","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"tsne_plot(cbow)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-13T17:35:02.420073Z","iopub.status.busy":"2020-10-13T17:35:02.417648Z","iopub.status.idle":"2020-10-13T17:35:53.256351Z","shell.execute_reply":"2020-10-13T17:35:53.256943Z"},"papermill":{"duration":52.344586,"end_time":"2020-10-13T17:35:53.257098","exception":false,"start_time":"2020-10-13T17:35:00.912512","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"tsne_plot(skipgram)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"execution":{"iopub.execute_input":"2020-10-13T17:35:56.189232Z","iopub.status.busy":"2020-10-13T17:35:56.186721Z","iopub.status.idle":"2020-10-13T17:36:05.400038Z","shell.execute_reply":"2020-10-13T17:36:05.39935Z"},"papermill":{"duration":10.693591,"end_time":"2020-10-13T17:36:05.400171","exception":false,"start_time":"2020-10-13T17:35:54.70658","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"question_vectors = list(zip(train['tags'].values, [cbow.get_word_vector(word) for word in train['tags']]))","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":1.532591,"end_time":"2020-10-13T17:36:08.446374","exception":false,"start_time":"2020-10-13T17:36:06.913783","status":"completed"},"tags":[]},"cell_type":"markdown","source":"**Disclaimer: I could be wildly off with my approach here, so I welcome thoughts and suggestions. If you see a mistake or have a question, just let me know.**"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}