{"cells":[{"metadata":{},"cell_type":"markdown","source":"# NLP Cheatsheet: Master NLP","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Since Kaggle has been recently awash of NLP competitions, I told myself that it would be a great opportunity to **share my knowledge by posting questions** (more or less advanced) on NLP topics.\n\nIt is a great way for you guys to test your knowledge of the field as well as to **understand better the underlying algorithms** of the famous frameworks that we use on a daily basis to model a problem.\n\nThis notebook is mostly a repertory of questions that can shape your learning of the field.\n\nAs a tip, I would recommend learning the structure (to have a structrued view of NLP) using [this notebook](https://www.kaggle.com/rftexas/ml-cheatsheet-a-mind-map-for-nlp) and to look for details when needed using this notebook.\n\nI will continually update this notebook regularly with questions that you can include in your favorite **note-taking** or **spaced-repetition** software (Anki is a must for this matter).\n\n<p style=\"text:red\"><b>If you like this work, don't forget to upvote the notebook as it encourages me to do it for many other fields. In particular, I plan on doing the same for Transformers, geometric deep learning and computer vision in the near future.</b></p>\n\nIf you want to have a more **holistic** approach to NLP, check out my other NLP notebook: https://www.kaggle.com/rftexas/ml-cheatsheet-a-mind-map-for-nlp","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Also this notebook mostly summarizes those amazing resources:\n- https://nlpoverview.com/#d-attention-mechanism\n- https://medium.com/saarthi-ai/transformers-attention-based-seq2seq-machine-translation-a28940aaa4fe\n- http://web.stanford.edu/class/cs224n/ (almost all the illustrations are extracted from this amazing course)\n- https://lilianweng.github.io/lil-log/2020/04/07/the-transformer-family.html\n- https://lilianweng.github.io/lil-log/2019/01/31/generalized-language-models.html","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Table of contents\n\n\n- <a href='#1'>1. Learning representations that conveys semantic and syntactic information</a>\n\n     - <a href='#1-1'>1.1. One-hot vector, a naïve approach (denotational semantics)</a>\n     - <a href='#1-2'>1.2. SVD-based methods (distributional semantics)</a>\n     \n         - <a href='#1.2.1'>1.2.1. Word-document matrix</a>\n         - <a href='#1.2.2'>1.2.2. Window-based co-occurence matrix</a>\n         \n     - <a href='#1-3'>1.3. Word2Vec</a>\n     - <a href='#1-4'>1.4. Global vector for word representation (GloVe)</a>\n\n\n- <a href='#2'>2. How to create language models?</a>\n\n     - <a href='#2-1'>2.1. N-gram language models</a>\n     - <a href='#2-2'>2.2. Recurrent Neural networks (RNN)</a>\n     - <a href='#2-3'>2.3. Deep bidirectional RNN</a>\n     - <a href='#2-4'>2.4. GRU and LSTM</a><br/>\n     \n     \n- <a href='#3'>3. How to deal with sequential output with Seq2Seq models?</a>\n\n     - <a href='#3-1'>3.1. Seq2Seq models</a>\n     - <a href='#3-2'>3.2. Attention mechanism</a>\n     \n     \n- <a href='#4'>4. How to deal with large vocabulary?</a>\n\n     - <a href='#4-1'>4.1. Scaling softmax</a>\n     - <a href='#4-2'>4.2. Word and character-based models</a>\n     \n         - <a href='#4.2.1'>4.2.1. Word segmentation</a>\n         - <a href='#4.2.2'>4.2.2. Character-based model</a>\n         - <a href='#4.2.3'>4.2.3. FastText embeddings</a>\n         - <a href='#4.2.4'>4.2.4. Hybrid NMT</a><br/>\n         \n         \n- <a href='#5'>5. How to create contextual embeddings?</a>\n\n     - <a href='#5-1'>5.1. ELMo</a>\n     - <a href='#5-2'>5.2. ULMFit</a>\n     - <a href='#5-3'>5.3. Transformer models</a>\n     \n         - <a href='#5.3.1'>5.3.1. OpenAI GPT</a>\n         - <a href='#5.3.2'>5.3.2. BERT</a>\n         - <a href='#5.3.3'>5.3.3. RoBERTa</a>\n         - <a href='#5.3.4'>5.3.4. ALBERT</a>\n         - <a href='#5.3.5'>5.3.5. ELECTRA</a>\n         - <a href='#5.3.6'>5.3.6. DistilBERT</a>\n         - <a href='#5.3.7'>5.3.7. XLNet</a>\n\n\n- <a href='#6'>6. Miscellaneous</a>","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 1. Learning representations that convey semantic and syntactic information\n\nA central problem in NLP is how to represent words as input to any of our models. We need some notions of similarity and distance between words. Intuitively, we do feel that queen is closer to king than cat for instance.\n\nThus, we want to encode word tokens into some vectors that represent points in some 'word space'.\n\n**Objective**: Finding a N-dimensional space (where N is between 100 and 1000) that is sufficient to encode all semantics of our language. In this word space, each dimension would encode some meaning that we transfer using speech (e.g. tense, count, gender...)\n\n\n- **What are the four main steps of text processing?**\n\n    1) **Loading**: Load text as string into memory \n\n    2) **Cleaning**: Cleaning the text, correct misspellings, remove emojis\n\n    3) **Tokenization**: Split strings into tokens, where a token could be a word or a character\n\n    4) **Vectorization**: Map all the tokens in data into vectors for ease of feeding into models\n\n\n- **What does bag of words refer to?**\n\n    When we put all the words of a document in a *'bucket'* we call such a bucket a bag of words. Simply put, a bag of words is a set of all the words in a document.\n\n\n- **What are stop words?**\n\n    Stop words are essentially high-frequency generic words that do not convey context-specific sense. E.g.: 'the', 'of', ...\n\n\n- **What is TF-IDF vectorizer?**\n\n    TF-IDF stands for term frequency - inverse data frequency. TF is the number of times a word appears in a document divided by the total number of words in the document. IDF is the log of the numbers of documents divided by the number of documents that contain the word w. TF-IDF is the product of those two quantities.\n\n\n- **What is a word embedding?**\n\n    Word embedding is a dense representation of words in the form of numeric vectors.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1.1. One-hot vector, a naïve approach (denotational semantics)\n\n- **What is one-hot word representation?**\n\n    Every word is represented as a V-dimensional vector (where V is the size of the vocabulary), with all 0s and 1 at the index of that word in the sorted English language.\n\n\n- **What is denotational semantics?**\n\n    The concept of representing an idea as a symbol. It is sparse and cannot capture similarity.\n    \n\n- **What is the issue of one-hot encoding?**\n\n    No notion of similarity (no cosine similarity for instance).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1.2. SVD-based methods (distributional semantics)\n\nObjective: Finding word embeddings that captures some notion of similarity.\n\n- **What is distributional semantics?**\n\n    The concept of representing the meaning of a word based on the context in which it usually appears. It is dense and can better capture similarity.\n\n- **How do SVD-based methods work?**\n\n    Loop over a massive dataset and accumulate word co-occurence counts in a matrix X. Then use SVD where the word vectors are the columns of U.\n\n- **What is latent semantic analysis (LSA)?**\n\n    LSA is a technique of analyzing relationships between a set of documents and the terms they contain by producing a set of concepts related to the documents and terms. To do so, we build a matrix whose coefficients are word counts per document and use SVD to reduce the number of rows while preserving the similarity structure among columns. To compare documents, we compute the cosine similarity between the column vectors representing them.\n\n\n### 1.2.1. Word-document matrix\n\n- **What is the assumption of a word-document matrix?**\n\n    We make the bold conjecture that words that are related will often appear in the same documents. For instance, *banks*, *bonds* and *stocks* are probably likely to appear together. But *banks*, *octopus* and *hockey* would probably not consistently appear together.\n\n\n- **How is a word-document matrix built?**\n\n    Loop over billions of documents and for each time word *i* appears in document *j* add one to entry *Xij*.\n\n\n- **What is the issue of a word-document matrix?**\n\n    A very large matrix that scales with the number of documents.\n\n\n### 1.2.2. Window-based co-occurrence matrix\n\n\n- **What is a window-based co-occurrence matrix?**\n\n    A matrix stores co-occurrences of words thereby becoming an affinity matrix. In this method, we count the number of times each word appears inside a window of a particular size around the word of interest.\n\n\n- **How to extract word vectors from a word-word co-occurrence matrix?**\n\n    Generate a VxV co-occurrence matrix X. Apply SVD on X. Select the first k columns of U to get a k-dimensional word vectors. The ratio of the variances indicates the amount of variance captured by the first k dimensions.\n\n\n- **What are the problems faced with such a method?**\n    - The dimensions of the matrix change very often\n    - The matrix is sparse\n    - Very high dimensional\n    - Quadratic cost to train","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1.3. Word2Vec\n\n- **What is an iteration-based model?**\n\n    A model that is able to learn one iteration at a time and eventually be able to encode the probability of a word given its context.\n\n\n- **What is Word2Vec?**\n\n    A model whose parameters are the word vectors. Train the model on a certain objective. At every iteration, we run our model, evaluate the errors and backpropagate the gradients in the model.\n\n\n- **What are the initial embeddings of Word2Vec model?**\n\n    The embedding matrix is initialized randomly using a Normal or uniform distribution. Then, the embedding of word *i* in the vocabulary is the row *i* of the embedding matrix.\n\n\n- **What are the two algorithms used by Word2Vec? Explain how they work.**\n\n    Continuous bag-of-words (CBOW)\n\n    Skip-gram\n\n\n- **What are the two training methods used?**\n\n    Hierarchical softmax\n\n    Negative sampling\n\n\n- **What is the advantage of Word2Vec over SVD-based methods?**\n\n    Much faster to compute and capture complex linguistic patterns beyond word similarity\n\n\n- **What is the limitation of Word2Vec?**\n\n    Fails to make use of global co-occurrence statistics. It only relies on local statistics (words in the neighborhood of word *i*). \n\n    E.g.: The cat sat on the mat. Word2Vec doesn't capture if *the* is a special word in the context of cat or just a stop word.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1.4. Global vectors for word representations (GloVe)\n\n- **What is GloVe?**\n\n    GloVe aims to combine the SVD-based approach and the context-based skip-gram model.\n\n- **How to build a co-occurence matrix for GloVe? What can we calculate with such a matrix?**\n\n    Let X be a word-word co-occurence matrix (coefficients are the number of times word *i* appears in the context of word *j*). With this matrix, we can compute the probability of word *i* appearing in the context of word j: *Pij = Xij / Xi*\n\n- **How is GloVe built?**\n\n    After building the co-occurence matrix, GloVe computes the ratios of co-occurrence probabilities (non-zero). The intuition is that the word meanings are capture by the ratios of co-occurrence probabilities rather than the probabilities themselves. The global vector models the relationship between two words regarding to the third context word as:\n\n    $$F(w_i, w_j, \\tilde{w}_k) = \\frac{p_{\\text{co}}(\\tilde{w}_k \\vert w_i)}{p_{\\text{co}}(\\tilde{w}_k \\vert w_j)}$$\n\n    F is designed to be a function of the linear difference between two words *wi* and *wj*. It is an exponential function.\n\n- **What are the pros of GloVe?**\n\n    The GloVe model efficiently leverages global statistical information by training only on non-zero elements in a word-word co-occurence matrix, and produces a vector space with meaningful substructure.\n\n- **What is window classification and why is it important?**\n\n    Natural languages tend to use the same word for very different meanings and we typically need to know the context of the word usage to discriminate between meanings.\n\n    E.g.: 'to sanction' means depending on the context 'to permit' or 'to punish'\n\n    A sequence is a central word vector preceded and succeeded by context word vectors. The number of words in the context is also known as the context window size and varies depending on the problem being solved.\n\n- **How do window size relate to performance?**\n\n    Generally, narrower window size lead to better performance in syntactic tests while wider windows lead to better performance in semantic tests.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 2. How to create language models?\n\n- **What are language models?**\n\n    Language models compute the probability of occurrence of a number of words in particular sequence.\n\n\n## 2.1. N-gram language models\n\nObjective: Compute the probability of occurrence of a number of words in particular sequence looking at the n previous words.\n\n\n- **What are the 2 limitations of n-gram language models?**\n    - **Sparsity problems**: if n-gram never appears in corpus, then probability is 0.\n    - **Storage problems**: as n increases or the corpus size increases, the model size increases as well.\n\nSince we only consider the n previous words in the sequence, we cannot take advantage of the full information conveyed in a sequence.\n\n## 2.2. Recurrent neural networks (RNN)\n\nObjective: Condition the language model on all previous words in the corpus.\n\n\n- **How is organized RNN?**\n\n    A RNN is organized in a series of hidden layers holding a number of neurons, each of which performs a linear matrix operation on its inputs followed by a non-linear operation.\n\n    At each time-step, there are two inputs to the hidden layer: the output of the previous layer *h(t-1)*, and the input at that timestep *x(t)*. \n\n    $$h_{t} = \\sigma(W^{(hh)}h_{t-1}+W^{(hx)}x_{t})$$\n\n    $$\\hat{y_{t}}=softmax(W^{(S)}h_{t})$$\n\n    After this operation, the output h(t) is multiplied by a weight matrix and run through a softmax over the entire vocabulary, which outputs the probabilities of the next word. The word with the highest probability is picked.\n\n\n- **How does a RNN solve the curse of dimensionality problem incurred by n-gram language models?**\n\n    It is solved since the weight matrices are applied at every step of the network. Hence the model parameters don't grow proportionally to the input sequence size. The number of parameters is independent of the sequence length.\n\n\n- **What is the loss function of a RNN?**\n\n    Cross-entropy summed over a corpus of size T and a vocabulary of size V.\n\n\n- **What is the perplexity of a RNN and what does it mean to have a low perplexity?**\n\n    The perplexity of a RNN is 2 to the negative log probability of the cross entropy loss function.\n\n    Perplexity is a measure of confusion where lower values imply more confidence in predicting the next word in the sequence.\n\n    A perplexity for a language model of 247 means that the model is as confused/perplex as if it has to choose uniformly and independently among 247 possibilities for each word.\n\n\n- **What are advantages of RNN?**\n    - They can process input sequences of any length\n    - The model size does not increase for longer input sequence lengths\n    - The same weights are applied at every time step of the input, so there is symmetry in how inputs are processed\n    \n    \n- **Give an example of the vanishing gradient problem in RNN and explain it.**\n\n    > S1: 'Jane walked into the room. John walked in too. Jane said hi to *?*'.\n\n    > S2: 'Jane walked into the room. John walked in too. It was late in the day, and everyone was walking home after a long day at work. Jane said hi to *?*'.\n\n    In both cases, the RNN should predict John as an answer. However, in practice, it turns out the RNN is more likely to predict John in sentence 1 than in sentence 2. Indeed, during backpropagation, the contribution of gradient values **gradually vanishes as they propagate to earlier timesteps**. Thus for long sentences, the RNN is less likely to recall information introduced in the earliest part of a sentence.\n\n\n- **How to solve vanishing gradient problem?**\n\n    **Technique 1**: Instead of initializing W randomly, start off from an identity matrix initialization.\n\n    **Technique 2**: Use ReLU as an activation function since the derivative of the gradient is either 0 or 1. This way, gradients would flow through the neurons whose derivatives is 1 without getting attenuated while propagating back through time-steps.\n\n\n- **What are exploding gradients and give a technique on how to solve them?**\n\n    The explosion occurs through exponential growth by repeatedly multiplying gradients through the network layers that have values larger than 1.0.\n\n    A technique to solve exploding gradients is gradient clipping. Gradient clipping is a simple heuristic invented by Thomas Mikolov to counter the effect of exploding gradient. That is, whenever the gradient reach a certain threshold, they are set back to a small number.\n\n\n## 2.3. Deep bidirectional RNN\n\nAssumption: It is possible to predict a word by looking at future words. We should be able to create more accurate and contextual representation of a word by concatenating forward and backward hidden layers.\n\n\n- **What is a deep bidirectional RNN?**\n\n    It is possible to make predictions based on future words in the sequence. The RNN should read forward and backward at the same time to make an accurate prediction at time t.\n\n    This network maintains two hidden layers, one for the left-to-right propagation and another for the right-to-left propagation.\n\n    $$\\overrightarrow{h_{t}} = f(\\overrightarrow{W}x_{t}+\\overrightarrow{V}\\overrightarrow{h_{t-1}}+\\overrightarrow{b})$$\n\n    $$\\overleftarrow{h_{t}} = f(\\overleftarrow{W}x_{t}+\\overleftarrow{V}\\overleftarrow{h_{t-1}}+\\overleftarrow{b})$$\n\n    $$\\hat{y_{t}}=g(U[\\overrightarrow{h_{t}};\\overleftarrow{h_{t}}] + c)$$\n\n\n- **What is a limitation of deep bidirectional RNN?**\n\n    Exploding and vanishing gradients.\n\n\n## 2.4. GRU and LSTM cells\n\n\n- **Why do we need GRU?**\n\n    Although RNNs can theoretically capture long-term dependencies, they are very hard to actually train to do this. GRU are designed in a manner to have more persistent memory thereby making it easier for RNNs to capture long-term dependencies.\n\n\n- **What are the 4 gates of a GRU cell?**\n    - New memory generation\n    - Reset gate\n    - Update gate\n    - Hidden state\n    \n    \n- **What are the new memory stage and reset gate in a GRU?**\n\n    The reset signal is responsible for determining how important h(t-1) is to the summarization of h(t).   The reset gate has the ability to completely diminish past hidden state if it finds that h(t-1) is irrelevant to the computation of the new memory.\n\n    $$r_{t}=\\sigma(W^{(r)}x_{t}+U^{(r)}h_{t-1})$$\n\n    This stage is the one who knows the recipe of combining a newly observed word with the past hidden state h(t-1) to summarize this new word in light of the contextual past.\n\n    $$h_{t}=tanh(r_{t}\\circ Uh_{t-1}+Wx_{t})$$\n\n\n- **What is the update gate in a GRU?**\n\n    The update gate is responsible for how much of h(t-1) should be carried forward to the next state.\n\n    $$z_{t}=\\sigma(W^{(z)}x_{t}+U^{(z)}h_{t-1})$$\n\n\n- **Give the equation of the new hidden state in a GRU cell.**\n\n    $$h_{t}=(1-z_{t})\\circ \\hat{h}_{t}+z_{t}\\circ h_{t-1}$$\n\n    where z is the update gate, h hat the new memory and h(t-1) the previous hidden state.\n\n\n- **What are the 6 memory cells of LSTM?**\n    - Input gate\n    - Forget gate\n    - Output/exposure gate\n    - New memory cell\n    - Final memory cell\n    \n \n- **What is the difference between GRU and LSTM cells?**\n\n    The GRU controls the flow of information like the LSTM unit, but without having a memory unit. It just exposes the full hidden content without any control.\n\n    GRU performance is on par with LSTM, but computationally more efficient (less complex structure as pointed out).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 3. How to deal with sequential output with Seq2Seq models?\n\nSo far, we have predicted a single output: an NER label for a word, the single most likely next word in a sentence given the past few. However, there's a whole class of NLP tasks that rely on sequential output.\n\nE.g.: translation, conversation, summarization\n\n\n## 3.1. Seq2Seq models\n\n- **What is the advantage of Seq2Seq models over regular RNNs?**\n\n    Seq2Seq models can generate arbitrary output sequences after seeing the entire input. They can even focus in on specific parts of the input automatically to help generate a useful translation.\n\n    E.g.: 1 encoder and 1 decoder (both LSTM or bi-LSTM)\n\n\n- **What does an encoder do in a Seq2Seq model?**\n\n    It reads the input sequence and generate a fixed-dimensional context vector C for the sequence.\n\n    The encoder stacks multiple RNNs on top of each other to effectively compress an arbitrary-length sequence into a fixed-size vector.  The final layer's LSTM hidden state will be used as a context vector.\n\n\n- **What does a decoder do in a Seq2Seq model?**\n\n    It uses the context vector as a *'seed'* from which to generate an output sequence.\n\n\n- **In which order does an encoder read a sentence?**\n\n    Reverse\n\n\n- **How does a decoder work in a Seq2Seq model?**\n\n    We initialize the hidden state of our first layer with the context vector, we then pass an <EOS> token appended to the end of the input. We then run the three-stacked RNN, following up with a softmax on the final layer's output to generate the first word.\n\n\n## 3.2. Attention mechanism\n\n\n- **Why do we need attention mechanism?**\n\n    Some words convey more information than others, hence requires more attention. Thus, there is a flaw in using the final RNN hidden state as a context vector: different parts of the input have different levels of significance. Moreover, different parts of the output may even consider different parts of the input *'important'*.\n\n    *E.g.*: The ball is on the field. \n\n    Most important words: ball, on, field.\n\n\n- **How does attention work?**\n\n    We compute an attention vector. It is a vector of weights that are used to compute the context vector as a weighted average of the hidden states generated by the encoder at time steps *1, ..., n*.\n\n    The decoder network is provided with a look at the entire input sequence at every decoding step since it is fed with the context vector. The decoder can then decide what input words are important.\n\n\n- **What is global attention?**\n\n    Instead of giving to the decoder's LSTM cells a single context vector which is a weighted average of all the hidden states of the encoder (attention). LSTM cells in the decoder are fed with the **concatenation** of the encoder hidden states at time i and the context vector.\n\n\n- **Give 4 algorithms that can be used by decoders to search translations.**\n\n    Exhaustive search, ancestral sampling, greedy search, beam search\n\n\n- **How does beam search work?**\n\n    We maintain K candidates at each time step. To do so, we compute H(t+1) by expanding H(t) and keeping the best K candidates (the ones with the highest probability).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 4. How to deal with large vocabulary?\n\nSeq2Seq models have a hard time dealing with large vocabulary size. These models predict the next word in the sequence by computing a target probabilistic distribution over the entire vocabulary using softmax. **Softmax is expensive to compute and its complexity scales proportionally to the vocabulary size**. \n\n## 4.1. Scaling softmax\n\n- **Give two techniques to scale softmax**.\n\n    Noise contrastive estimation and hierarchical softmax\n\n\n- **What is the issue with both techniques?**\n\n    Both methods save computation during training step (when target word is known). At test time, one still has to compute the probability of all words in the vocabulary in order to make predictions.\n\n## 4.2. Word and character-based models\n\nPhonology posits a small set or sets of distinctive, categorical units that we call **phonemes**. Let's try to change our perspective: instead of working with words or even splitting words into characters, let's now consider **n-gram characters**.\n\n- **Name three families of techniques to deal with unknown or rare words.**\n\n    Word segmentation (adaptation of Byte-pair encoding), character-based model, hybrid model\n\n\n### 4.2.1. Word segmentation\n\n- **How does word segmentation solve the issue related to large vocabulary?**\n\n    Word segmentation represents unknown or rare words as a sequence of subword units.\n\n\n- **What is Byte-pair encoding?**\n\n    Byte-pair encoding is a lossless compression algorithm. In NLP, it is adapted to transform rare words into character tokens. \n\n    For instance: athazagoraphobia becomes ['_ath', 'az', 'agor', 'aphobia']\n\n    It follows the following procedure:\n\n    1. Represent each word in the corpus as a combination of the characters along with the special end of word token </w> \n    2. Iteratively count character pairs in all tokens of the vocabulary\n    3. Merge every occurrence of the most frequent pair, add the new character n-gram to the vocabulary\n    4. Repeat step 3 until the desired number of merge operations are completed or the desired vocabulary size is achieved\n\n    > {'l o w </w>': 5, 'l o w e r </w>': 2, 'n e w e s t </w>': 6, 'w i d e s t </w>': 3}\n\n    > {'l o w </w>': 5, 'l o w e r </w>': 2, 'n e w es t </w>': 6, 'w i d es t </w>': 3}\n\n    > {'l o w </w>': 5, 'l o w e r </w>': 2, 'n e w est </w>': 6, 'w i d est </w>': 3}\n\n\n- **How does word segmentation work?**\n\n    Start with a vocabulary of characters and keep extending the vocabulary with most frequent n-gram pairs in the data set. This process is repeated until all n-gram pairs are selected or vocabulary size reaches some threshold.\n\n\n- **What is the difference between WordPiece and SentencePiece?**\n\n    WordPiece is an algorithm that creates smaller units than words (sub-word level).\n\n    SentencePiece - created by Google - is an algorithm that combines sub-word level tokenization (BPE) as well as unigram tokenization.\n\n\n### 4.2.2. Character-based models\n\n\n- **How are character-level embeddings generated?**\n\n    Character-level embeddings use one-dimensional CNN to find numeric representation of words by looking at their character-level compositions.\n\n\n### 4.2.3. FastText embeddings\n\n- **Give a brief explanation of FastText embeddings.**\n\n    Simply put, FastText is a word2vec-like algorithm that leverages n-gram word pieces to encode a large and open vocabulary and to capture word morphology.\n\n\n- **How are words represented in FastText?**\n\n    They are represented as char n-grams augmented with boundary symbols and as whole word.\n\n    E.g.: where = <wh, whe, her, ere, re>, < where >\n\n\n- **How are FastText embeddings generated?**\n\n    A bi-LSTM is trained to compute embeddings.\n\n\n### 4.2.4. Hybrid NMT\n\n\n- **How does a hybrid Neural Machine Translation work?**\n\n    The system translates mostly at word-level and consults the character components for rare words.\n\n\n- **Give the structure of a hybrid NMT.**\n\n    A word-based translation as a backbone\n\n    A source character-based representation\n\n    A target character-level generation\n\n\n- **What is the purpose of a word-based translation as a backbone in a hybrid NMT?**\n\n    This is the core of a hybrid NMT (LSTM encoder-decoder) that translates at the word level. We use <unk> to represent OOV words.\n    \n\n- **What is the purpose of a source character-based representation model in a hybrid NMT?**\n\n    A deep LSTM model that learns over characters of rare words and use the final hidden state of the LSTM as the representation for the rare word.\n    \n\n- **What is the purpose of a target character-level generation model?**\n\n    The goal is to create a coherent representation that handles unlimited output vocabulary. To do so, we have a separate deep LSTM that *'translates'* at the character-level given the current word-level state.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 5. How to create contextual word embeddings?\n\nSo far, we have always the same representation for a word type regardless of the context in which a word token occurs. Word embeddings are context-free with GloVe, Word2Vec or FastText. Those language models are trained to predict the next word but they are producing context-specific word representations at each position.\n\nE.g.: I ate an apple. I have an Apple iPhone. \n\nIn both cases, the word *'apple'* shares the same word embedding. However, they mean two radically different things. \n\n\n## 5.1. ELMo (Embeddings from language model)\n\n- **What is ELMo?**\n\n    ELMo learns contextualized word representation by pre-training a language model in an unsupervised way.\n\n\n- **How is ELMo trained?**\n\n    The bidirectional language model is the foundation of ELMo. While the input is a sequence of **n** tokens, the language model learns to predict the probability of next token given the history.\n\n    **Forward pass:**\n\n    $$p(x_1, \\dots, x_n) = \\prod_{i=1}^n p(x_i \\mid x_1, \\dots, x_{i-1})$$\n\n    **Backward pass:**\n\n    $$p(x_1, \\dots, x_n) = \\prod_{i=1}^n p(x_i \\mid x_{i+1}, \\dots, x_n)$$\n\n    The model is trained to minimize the negative log-likelihood in both directions:\n\n    $$\\begin{aligned} \\mathcal{L} = - \\sum_{i=1}^n \\Big(  \\log p(x_i \\mid x_1, \\dots, x_{i-1}; \\Theta_e, \\overrightarrow{\\Theta}_\\text{LSTM}, \\Theta_s) + \\\\ \\log p(x_i \\mid x_{i+1}, \\dots, x_n; \\Theta_e, \\overleftarrow{\\Theta}_\\text{LSTM}, \\Theta_s) \\Big) \\end{aligned}$$\n\n\n- **How does ELMo learn task-specific representations?**\n\n    On top of a L-layer biLM, ELMo stacks all the hidden states across layers together by learning a task-specific linear combination.\n\n    The weights, s_task, in the linear combination are learned for each end task and normalized by softmax. The scaling factor γtask is used to correct the misalignment between the distribution of biLM hidden states and the distribution of task specific representations.\n\n    $$v_i = f(R_i; \\Theta^\\text{task}) = \\gamma^\\text{task} \\sum_{\\ell=0}^L s^\\text{task}_i \\mathbf{h}_{i,\\ell}$$\n\n\n- **To which tasks correspond which layers?**\n\n    The comparison study indicates that syntactic information is better represented at lower layers while semantic information is captured by higher layers. Because different layers tend to carry different type of information, **stacking them together helps**.\n\n\n## 5.2. ULMFiT\n\n- **What is innovative with ULMFiT?**\n\n    ULMFiT is the first model to introduce the idea of generative pretrained language models that is fine-tuned for a specific task. \n\n\n- **What are the three steps to achieve good transfer learning results?**\n    1. General LM pre-training (already done by [fast.ai](http://fast.ai) researchers) on Wikipedia\n    2. Target task LM fine-tuning: finetuning LM on a specific vocabulary\n    3. Train a target task classifier with 2 fully-connected layers\n\n- **What are the two techniques used when fine-tuning LM?**\n    1. **Discriminative fine-tuning**: tune each layer with different learning rates\n    2. **Slanted triangular learning rates**: triangular learning rate schedule\n    \n- **What are the two techniques used when fine-tuning the classifier?**\n    1. **Concat pooling** extracts max-polling and mean-pooling over the history of hidden states and concatenates them with the final hidden state.\n    2. **Gradual unfreezing** extracts max-polling and mean-pooling over the history of hidden states and concatenates them with the final hidden state.\n\n\n## 5.3. Transformers\n\n### 5.3.1. OpenAI GPT\n\n- **What does GPT stand for?**\n\n    Generative Pre-training transformer\n\n\n- **What are the two major difference between ELMo and OpenAI GPT?**\n    1. The model architectures are different: ELMo uses a shallow concatenation of independently trained left-to-right and right-to-left multi-layer LSTMs, while GPT is a multi-layer transformer decoder.\n    2. ELMo feeds embeddings into models customized for specific tasks as additional features, while GPT fine-tunes the same base model for all end tasks.\n    \n    \n- **What is the major upgrade brought by OpenAI GPT?**\n\n    To get rid of the task-specific model and use the pre-trained language model directly.\n\n    Hence we don't need new a new design for specific tasks. We just need to modify the input sequence by adding custom tags. At the first stage, generative pre-training on a language model can absorb as much free text as possible. Then at the second stage, the model is fine-tuned on specific tasks with a small labeled dataset and a minimal set of new parameters to learn.\n\n\n- **What is the loss of OpenAI GPT?**\n\n    The loss is the negative log-likelihood for true labels to which we add the LM loss as an auxiliary loss. \n\n    $$\\mathcal{L}_\\text{cls} = \\sum_{(\\mathbf{x}, y) \\in \\mathcal{D}} \\log P(y\\mid x_1, \\dots, x_n) = \\sum_{(\\mathbf{x}, y) \\in \\mathcal{D}} \\log \\text{softmax}(\\mathbf{h}_L^{(n)}(\\mathbf{x})\\mathbf{W}_y)$$\n\n    $$\\mathcal{L}_\\text{LM} = -\\sum_{i} \\log p(x_i\\mid x_{i-k}, \\dots, x_{i-1})$$\n\n    $$\\mathcal{L} = \\mathcal{L}_\\text{cls} + \\lambda \\mathcal{L}_\\text{LM}$$\n\n\n- **What is a limitation of OpenAI GPT?**\n\n    Its unidirectional nature.\n\n\n### 5.3.2. BERT\n\n- **What is the biggest difference between OpenAI GPT and BERT? Which limitation does it solve?**\n\n    OpenAI is unidirectional. BERT is bidirectional.\n\n\n- **What is the base component of BERT?**\n\n    The model architecture of BERT is a multi-layer bidirectional Transformer encoder. It is composed of Multi-headed self attention, feed-forward layers, layer norm and residuals and positional embeddings.\n\n\n- **What are the tasks on which BERT is trained?**\n\n    1. **Mask language model (MLM)**: Randomly mask 15% of tokens in each sequence. \n\n        BERT employed several heuristic tricks:\n\n        - (a) with 80% probability, replace the chosen words with `[MASK]`;\n        - (b) with 10% probability, replace with a random word;\n        - (c) with 10% probability, keep it the same.\n        \n    2. **Next sentence prediction**: tell whether one sentence is the next sentence of the other. It is used to learn relationships between sentences.\n    \n    \n- **Give the 3 components that constitutes BERT input embedding.**\n\n    1. WordPiece embeddings (cf. previously)\n    2. Segment embeddings\n    3. Position embeddings (are learned)\n\n\n- **What is the structure of BERT base?**\n\n    12 layers, 768-dimensional output hidden state, 12 heads\n\n\n- **What is the structure of BERT large?**\n\n    24 layers, 1024-dimensional output hidden state, 16 heads\n\n\n- **What is the [SEP] token used for?**\n\n    [SEP] token is used when building a sequence from multiple sequences\n\n    E.g.: two sequences for sequence classification or for a text and a question for question answering\n\n\n- **What is the [PAD] token used for?**\n\n    The token used for padding, for example when batching sequences of different lengths\n\n\n- **What is the [CLS] token used for?**\n\n    The classifier token which is used when doing sequence classification. It is the first token of the sequence when built with special tokens.\n\n\n### 5.3.3. RoBERTa\n\n- **What does RoBERTa stand for?**\n\n    Robustly optimized BERT approach\n\n\n### 5.3.4. ALBERT\n\n- **What are the 3 innovations brought by ALBERT?**\n    1. Factorized Embedding parametrization\n    2. Cross-layer parameter sharing\n    3. Sentence order prediction\n    \n    \n- **What is factorized embedding parametrization?**\n\n    In BERT, the WordPiece tokenization embedding size ***E*** is configured to be the same as the hidden state size ***H***. That is saying, if we want to increase the model size (larger ***H***), we need to learn a larger tokenization embedding too, which is expensive because it depends on the vocabulary size (***V***).\n\n    Conceptually, because the tokenization embedding is expected to learn *context-independent* representation and the hidden states are *context-dependent*, it makes sense to separate the size of the hidden layers from the size of vocabulary embedding. Using factorized embedding parameterization, the large vocabulary embedding matrix of size ***V×H*** is decomposed into two small matrices of size ***V×E*** and ***E×H***.\n\n### 5.3.5. ELECTRA\n\n### 5.3.6. DistilBERT\n\n- **How does distillation work?**\n\n    - Train \"Teacher\": Use SOTA pre-training + fine-tuning technique to train model with maximum accuracy\n\n    - Label a large amount of unlabeled input examples with Teacher\n\n    - Train \"Student\": much smaller model which is trained to mimic Teacher output\n\n    - Student objective is typically MSE or cross-entropy\n\n### 5.3.7. XLNet\n\n- **What are two innovations of XLNet?**\n\n    1. Relative position embeddings\n    2. Permutation language modelling","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 6. Miscellaneous\n\n- **Explain what is self-supervised learning.**\n\n    Given a task and enough labels, supervised learning can solve it really well. Good performance usually requires a decent amount of labels, but collecting manual labels is expensive (i.e. ImageNet) and hard to be scaled up. Considering the amount of unlabelled data (e.g. free text, all the images on the Internet) is substantially more than a limited number of human curated labelled datasets, it is kinda wasteful not to use them. However, **unsupervised learning is not easy and usually works much less efficiently than supervised learning.**\n\n    **Self-supervised learning consists in getting labels for free for unlabelled data and train unsupervised dataset in a supervised manner**.\n    \n\n- **What is a problem induced by ReLU activations?**\n\n    Exploding gradients, dying ReLU, mean and variance of activations is not 0 or 1\n\n\n- **How to solve exploding gradients?**\n\n    Gradient clipping\n\n\n- **What is dying ReLU?**\n\n    No learning if the activation is 0\n\n\n- **How to solve dying ReLU?**\n\n    Parametric ReLU\n\n\n- **What is the difference between learning latent features using SVD and getting embedding vectors from a neural network?**\n\n    SVD uses linear combination of inputs while a neural network uses non-linear combination.\n\n\n- **Give the time complexity of LSTM.**\n\n    $$seqlength * hiddensize^{2}$$\n\n\n- **Give the time complexity of a transformer.**\n\n    $$seqlength^{2} * hiddensize$$\n\n\n- **How is AdamW different from Adam?**\n\n    AdamW is Adam with L2 regularization on weight as models with smaller weights generalise better\n\n\n- **What is the difference between LayerNorm and BatchNorm?**\n\n    BatchNorm: compute the mean and variance at each layer for every minibatch\n\n    LayerNorm: compute the mean and variance for every single sample for each layer independently\n\n\n- **What changes would you make to your deep learning code if you knew there are errors in your training data?**\n\n    Label smoothing where the smoothening values is based on % error. \n\n    Use class weights to modify the loss","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}