{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1ce4c313-f9bb-e99f-c980-ac49128a6690"
      },
      "source": [
        "GloVe is a database with a vector of 2.2 million of words\n",
        "-----------\n",
        "link [Stanford][1]\n",
        "\n",
        "Now what is this script doing ? See what a simple script, the beauty is always when its simple.. ;-)\n",
        "\n",
        " 1. Import the GloVe, 10 rows only, they have 100, 200, 300 vectorrows for the people with more powerful computers, the complete glove is 2Gig... I presume with the 'transfer learning' the aim of this contest is that you build up your own word database like this GloVe. How would you construct that database ? Simple: tfidf all the words, and vectorise with SVD, taking the Vt... I have done hit some notebooks before see [link][2]. So actually i replace the Vt_ with GloVe. Now to be honest i touch the limits of these servers here, they hang more than doing that job... I can't import that database here, i am working on my probably obsolete 4years old singlecored 32bit computer, so this game is also a matter of speed.\n",
        "  \n",
        " 2. Tfidf the title and content with the vocabulary of the Glove. top 400.000 words, gives you a very sparse matrix...\n",
        "\n",
        " 3. Find the good keywords.. What most people are doing with tfidf is simply selecting the most bizar words. Thats not bad, but thats not always the most relevant words in the context. You could use the ranking already of GloVe to find the same thing.\n",
        " Now to do the 'transferlearning' IMHO you need to calculate the similarity between all words in the text. Using that Vt_ information or the Glove vector information. What is happening is that the 'co-grouped' words can be selected.\n",
        "\n",
        "That is my first 10 results...\n",
        "\n",
        " - 1  spin particles subatomic \n",
        " - 2  how needed theory prove explanation\n",
        "   plausible simplest \n",
        " - 3  group groups \n",
        " - 7  \n",
        " - 9  principle integral \n",
        " - 13  \n",
        " - 15  string disprove \n",
        " - 17  blue color sky \n",
        " - 19  higher energy aim calculated  particles \n",
        " - 21  carlo monte\n",
        "\n",
        "  \n",
        "  \n",
        "\n",
        "\n",
        "  [1]: https://nlp.stanford.edu/projects/glove/\n",
        "  [2]: https://www.kaggle.com/plarmuseau/transfer-learning-on-stack-exchange-tags/theoretical-transfer-learning-script"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "be1d6888-260e-d3ed-6a1e-b4df2fc6a29b"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "# 1  Import glove 10 rows only , i lack memory and speed guys...\n",
        "Vt_=pd.read_table(\"../input/glove.6B.50d.txt\",usecols=[0,1,2,3,4,5,6,7,8,9,10,11],header=None, delim_whitespace=1,quoting=3)\n",
        "Vt_.columns=['word',1,2,3,4,5,6,7,8,9,10,11]\n",
        "\n",
        "#import test database\n",
        "test = pd.read_csv(\"../input/testw.csv\")\n",
        "test['category'] = 'physics'\n",
        "\n",
        "# 2  Tfidf vectorize with the glove.\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "tfidf3_vectorizer = TfidfVectorizer(vocabulary=Vt_['word'])\n",
        "D_ = tfidf3_vectorizer.fit_transform(test['title']+'. '+test['content'])\n",
        "# 82000*400k woorden 574000 words classified compressed in sparse matrix 300mb\n",
        "from scipy.sparse import csr_matrix\n",
        "D_=csr_matrix(D_)\n",
        "Vword=Vt_['word']\n",
        "Vt_=Vt_.drop('word', axis=1)\n",
        "\n",
        "# 3 Find the most relevant words...\n",
        "for xi in range (0,10):\n",
        "    idnr=test.ix[xi].id\n",
        "    Dtemp=pd.DataFrame(D_[xi,:].todense())\n",
        "    Dtemp=Dtemp.append(Vword.T)\n",
        "    Dtemp=Dtemp.append(Vt_.T)\n",
        "    Dtemp=Dtemp.T\n",
        "    Dtemp.columns=['rf','word','v1','v2','v3','v4','v5','v6','v7','v8','v9','v10','v11']\n",
        "    Dtemp.sort_values(by='rf')\n",
        "    Dbis=Dtemp[Dtemp['rf']>0]\n",
        "    maxrf=Dbis.max(axis=0).rf/2.2\n",
        "    Dbis=Dtemp[Dtemp['rf']>maxrf]\n",
        "    Dbis=Dbis.drop(['rf','word'],axis=1)\n",
        "    Dcorr=Dbis.dot(Dbis.T)/(abs(Dbis).dot(abs(Dbis.T)))\n",
        "    lengte=len(Dtemp[Dtemp['rf']>0])\n",
        "    corrm=Dcorr[0:lengte]\n",
        "    corrm=corrm.fillna(value=0)\n",
        "    superwoord=corrm[corrm>0.85].sum()\n",
        "    prnttmp=''\n",
        "    for wn in superwoord[superwoord>1].index:\n",
        "        prnttmp+=' '+Dtemp.ix[wn].word\n",
        "    print(idnr,prnttmp)"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}