{"cells":[{"metadata":{},"cell_type":"markdown","source":"Ref: https://www.youtube.com/watch?v=mmUttpXu7oE","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Theory\ndata imbalancing handling - \n* generating fake data with GANS\n* weighted sampling\n\nWe using model-roBERTa\ntakes text->word vector(768 D)(represent meaning and context)\n\nHow RoBERTa works-\nTransformers - Self attention: \nMy dog is lazy and it crossed the street(sentence) \nHow \"it\" refers to \"dog\"? -- attention\ndef: takes a bunch of words, understand context behind each words and makes sense about surrounding, massive deep neural network, with bunch of blocks in it, blocks are built around self attention, blocks are ENCODERS\n\nBERT - a transformer\nmost language model looks from left to right or right to left\nIn BERT-Bidirectional Encoder Representation of Transformers\n\"My name is Manami\" \nlooks at all these words at the same time \n* Huge corpus of text\n* took 15% of the words and masked them\n* based on other words, predicted the masked word - MLM\n* NSP - next sentence prediction\n    * gave sentences\n    * given- pair of 2 senences-second sentence directly follows the first(50% of the time)\n    * model predicts 0/1 (linked/notlinked)\n    * My name is Manami. I have a dog. His name is Blue.(2nd and 3rd sentences are linked)\n* BERT - trained on NSP, MLM\n* Fine tuning\n\nRoBERTa-\n* direclty based on BERT\n* NSP was removed\n* only trained on MLM\n* trained on much more data than BERT\n    ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Modeling pipeline\nQuestion text -> RoBERTa *o/p vector(meaning of the question)* ->Fully connected Dense Layer(1 neuron) ->sigmoid layer(prob 0-1)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Set up PyTorch-XLA**\nPytorch but on TPU\n* These few lines of code sets up PyTorch XLA for us.\n* We need PyTorch XLA to help us train PyTorch models on TPU.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Set the pytorch XLA TPU","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}