{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport io","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cce27e2b2c9458ec04de48b1ce1058cc271b1c6c"},"cell_type":"markdown","source":"**Competition rules in the nutshell:**\n\n* This competition does not allow external data.\n* Be aware that this is being run as a Kernels Only Competition, requiring that all submissions be made via a Kernel output.\n* Both your training and prediction should fit in a single Kernel. \n* GPUs are enabled for this competition. If you use GPUs, you will be limited to 2 hours of run time. If you do not use GPUs, you will be limited to 6 hours of run time. \n* No internet access enabled\n* No multiple data sources enabled\n* No custom packages\n* Submission file must be named \"submission.csv\""},{"metadata":{"_uuid":"ec86c48bc43cde91dd9f6db7e20ddde625071edd"},"cell_type":"markdown","source":"**Competition goal**\n> In this competition you will be predicting whether a question asked on Quora is sincere or not.\n"},{"metadata":{"_uuid":"9507fd957a2a01b55b72df66a4e47eb2990401f6"},"cell_type":"markdown","source":"**Data that you can use:**\n\n1. train.csv \n2. test.csv\n3. sample_submission.csv\n4. embeddings:\n    *  google\n    *  glove\n    *  paragram\n    * wiki"},{"metadata":{"_uuid":"fd3540c790040c1da6e3739a9ed26eab596d39ec"},"cell_type":"markdown","source":"**1. Train **\n\n*train.csv*"},{"metadata":{"trusted":true,"_uuid":"44741c8e2cd8579adde6b2271fd00801b1158ff4"},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\")\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"705cac231f7e3756718703c3e5c8b12b592d575f","trusted":true},"cell_type":"code","source":"#print((train['target'] == 1).sum())\n#print((train['target'] == 0).sum())\n\nx = np.arange(2)\nvalues = [(train['target'] == 0).sum(), (train['target'] == 1).sum()]\nfig, ax = plt.subplots()\nplt.bar(x, values)\nplt.xticks(x, ('0', '1'))\nplt.title('target column in train.csv')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"77f1de6e98be6c7a7a0068012f6571c2f8a3b4db"},"cell_type":"markdown","source":"Let's see sample questions that attack the rules and are marked as 1:"},{"metadata":{"trusted":true,"_uuid":"f0e83d8b9f7f5fab3a2154e0d19fbd64285e4812"},"cell_type":"code","source":"pd.set_option('display.max_colwidth', -1)\ntrain[train['target'] == 1].sample(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd65cca442bedae91c5dc038e2ec5830a34e4f60"},"cell_type":"markdown","source":"And sample properly formed questions:"},{"metadata":{"trusted":true,"_uuid":"f8b707a0e4261a788ecf2280baa4562d296c465e"},"cell_type":"code","source":"pd.set_option('display.max_colwidth', -1)\ntrain[train['target'] == 0].sample(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"33ad80cf1f379b883ac202872d6d60e28c335b51"},"cell_type":"markdown","source":"**4. Embeddings**\n\nTo start working with the data you can use this functions to read files:"},{"metadata":{"_uuid":"46d3379968abe5396c67007657aae06c575ac7a7"},"cell_type":"markdown","source":"\n*GoogleNews-vectors-negative300*\n\nUsage:"},{"metadata":{"trusted":true,"_uuid":"eba75abde9b962677b8e37685df12060de4f2ce9"},"cell_type":"code","source":"from gensim.models.keyedvectors import KeyedVectors  #import this to read binary file, isn't it against rule \"No custom packages\" ?\n\ndef loadGoogleModel(pathToFile):\n    googleModel = KeyedVectors.load_word2vec_format(pathToFile, binary=True)\n    return googleModel\n\npathToGoogleFile = \"../input/embeddings/GoogleNews-vectors-negative300/GoogleNews-vectors-negative300.bin\"\n\n# googleModel = loadGoogleModel(pathToGoogleFile)\n# result = googleModel.most_similar(positive=['dog'], topn=5) #you can also put the negative words ex. negative=['cat'], topn - number of top examples in return\n# print(result)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b2a8345ca088ba9ba09918d20ea1dc955b5a2bcb"},"cell_type":"markdown","source":" *GloVe: Global Vectors for Word Representation*\n\nUsage:"},{"metadata":{"_kg_hide-output":false,"trusted":true,"_uuid":"94da34784d69c89b357b1e839484e401d8031232"},"cell_type":"code","source":"def loadGloveModel(pathToFile):\n    print(\"Loading Glove Model\")\n    f = open(pathToFile,'r')\n    model = {}\n    for line in f:\n        splitLine = line.split(' ')\n        word = splitLine[0]\n        embedding = np.array([float(val) for val in splitLine[1:]])\n        model[word] = embedding\n    print(\"Done, words loaded!\")\n    return model\n\npathToGloveFile = \"../input/embeddings/glove.840B.300d/glove.840B.300d.txt\"\n\n# gloveModel = loadGloveModel(pathToGloveFile)\n# print(gloveModel['frog'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7445be728f04c7beb9d89627fb54c7de2ceb0e93"},"cell_type":"markdown","source":"*Paragram embeddings*\n\npretrained model from Cognitive Computation Group, Univeristy Pensylvania \n\nUsage:"},{"metadata":{"trusted":true,"_uuid":"9bcf18a32b3b2f27c629a56397c08709687ee2fd"},"cell_type":"code","source":"def loadParagramModel(pathToFile):\n    print(\"Loading Paragram Model\")\n    f = open(pathToFile,'r')\n    model = {}\n    for line in f:\n        splitLine = line.split(' ')\n        word = splitLine[0]\n        embedding = np.array([float(val) for val in splitLine[1:]])\n        model[word] = embedding\n    print(\"Done, words loaded!\")\n    return model\n\npathToParagramFile = \"../input/embeddings/glove.840B.300d/glove.840B.300d.txt\"\n\n# paragramModel = loadParagramModel(pathToParagramFile)\n# print(paragramModel['frog'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"31f202138dd77aa5c7fd652aeba990259fec8e3a"},"cell_type":"markdown","source":"*English word vectors: wiki-news*\n\nUsage:"},{"metadata":{"trusted":true,"_uuid":"018546950ee7ad5532958f739126e68264b34a30"},"cell_type":"code","source":"def loadWikiModel(pathToFile):\n    print(\"Loading Wiki Model\")\n    f = open(pathToFile,'r')\n    model = {}\n    for line in f:\n        splitLine = line.split(' ')\n        word = splitLine[0]\n        embedding = np.array([float(val) for val in splitLine[1:]])\n        model[word] = embedding\n    print(\"Done, words loaded!\")\n    return model\n\npathToWikiFile = \"../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec\"\n\n# wikiModel = loadWikiModel(pathToWikiFile)  \n# print(wikiModel['frog'])\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2843a6342bf4d34ca2b9ff2a1f55707dc57c9d9f"},"cell_type":"markdown","source":"**I will probably proceed to develop this kernel soon. Good luck!** "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}