{
  "id": 57299,
  "title": "Attention Layer for NN ",
  "url": "/competitions/avito-demand-prediction/discussion/57299",
  "author_name": "",
  "post_date": "2018-05-22T09:11:24.424847700Z",
  "votes": 2,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Can someone point me to a starter kernel for using Attention layers in RNN/CNN ? Any related online links to understand the concept will also be helpful.</p>\n\n<p>Thanks \nShanth</p>",
  "messages": [
    {
      "id": "331968",
      "postDate": "05/22/2018 09:11:24",
      "content": "<p>Can someone point me to a starter kernel for using Attention layers in RNN/CNN ? Any related online links to understand the concept will also be helpful.</p>\n\n<p>Thanks \nShanth</p>",
      "rawMarkdown": "Can someone point me to a starter kernel for using Attention layers in RNN/CNN ? Any related online links to understand the concept will also be helpful.\n\nThanks \nShanth",
      "votes": null
    },
    {
      "id": "331972",
      "postDate": "05/22/2018 09:24:42",
      "content": "<p>kernel: <a href=\"https://www.kaggle.com/sanghan/attention-with-fasttext-embeddings\">attention-with-fasttext-embeddings</a></p>\n\n<p>some reading (altough it is hierarchical attention) <a href=\"http://www.cs.cmu.edu/~./hovy/papers/16HLT-hierarchical-attention-networks.pdf\">hierarchical-attention-networks.pdf</a></p>",
      "rawMarkdown": "kernel: [attention-with-fasttext-embeddings][1]\n\nsome reading (altough it is hierarchical attention) [hierarchical-attention-networks.pdf][2]\n\n\n  [1]: https://www.kaggle.com/sanghan/attention-with-fasttext-embeddings\n  [2]: http://www.cs.cmu.edu/~./hovy/papers/16HLT-hierarchical-attention-networks.pdf",
      "votes": null
    },
    {
      "id": "331974",
      "postDate": "05/22/2018 09:25:27",
      "content": "<p>Thank you so much</p>",
      "rawMarkdown": "Thank you so much",
      "votes": null
    },
    {
      "id": "331987",
      "postDate": "05/22/2018 09:58:06",
      "content": "<p>I just added <a href=\"https://gist.github.com/cbaziotis/6428df359af27d58078ca5ed9792bd6d#file-attention-py\">this simple attention</a> after the RNN Layer . Beware of the training time ( if ever you don't have access to GPU)</p>\n\n<p>Hierarchical  attention  did not help that much and increased futhermore the training time (compared to simple Atention) </p>",
      "rawMarkdown": "I just added [this simple attention][1] after the RNN Layer . Beware of the training time ( if ever you don't have access to GPU)\n\nHierarchical  attention  did not help that much and increased futhermore the training time (compared to simple Atention) \n\n\n  [1]: https://gist.github.com/cbaziotis/6428df359af27d58078ca5ed9792bd6d#file-attention-py",
      "votes": null
    },
    {
      "id": "331990",
      "postDate": "05/22/2018 10:03:23",
      "content": "<p>Thanks Seringe</p>",
      "rawMarkdown": "Thanks Seringe",
      "votes": null
    },
    {
      "id": "343401",
      "postDate": "06/15/2018 09:24:16",
      "content": "<p>Hey Serigne, did attention provide a nice boost for you? Cheers.</p>",
      "rawMarkdown": "Hey Serigne, did attention provide a nice boost for you? Cheers.",
      "votes": null
    },
    {
      "id": "343969",
      "postDate": "06/16/2018 16:00:51",
      "content": "<p>Hi den3b...</p>\n\n<p>Sorry for the late reply </p>\n\n<p>Yes it provided some nice boost. </p>",
      "rawMarkdown": "Hi den3b...\n\nSorry for the late reply \n\nYes it provided some nice boost.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 331972,
      "author_name": "christofhenkel",
      "author_url": "",
      "post_date": "05/22/2018 09:24:42",
      "content": "<p>kernel: <a href=\"https://www.kaggle.com/sanghan/attention-with-fasttext-embeddings\">attention-with-fasttext-embeddings</a></p>\n\n<p>some reading (altough it is hierarchical attention) <a href=\"http://www.cs.cmu.edu/~./hovy/papers/16HLT-hierarchical-attention-networks.pdf\">hierarchical-attention-networks.pdf</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 331974,
          "author_name": "shanth84",
          "author_url": "",
          "post_date": "05/22/2018 09:25:27",
          "content": "<p>Thank you so much</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 331987,
      "author_name": "serigne",
      "author_url": "",
      "post_date": "05/22/2018 09:58:06",
      "content": "<p>I just added <a href=\"https://gist.github.com/cbaziotis/6428df359af27d58078ca5ed9792bd6d#file-attention-py\">this simple attention</a> after the RNN Layer . Beware of the training time ( if ever you don't have access to GPU)</p>\n\n<p>Hierarchical  attention  did not help that much and increased futhermore the training time (compared to simple Atention) </p>",
      "votes": null,
      "replies": [
        {
          "id": 331990,
          "author_name": "shanth84",
          "author_url": "",
          "post_date": "05/22/2018 10:03:23",
          "content": "<p>Thanks Seringe</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 343401,
          "author_name": "den3b81",
          "author_url": "",
          "post_date": "06/15/2018 09:24:16",
          "content": "<p>Hey Serigne, did attention provide a nice boost for you? Cheers.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 343969,
          "author_name": "serigne",
          "author_url": "",
          "post_date": "06/16/2018 16:00:51",
          "content": "<p>Hi den3b...</p>\n\n<p>Sorry for the late reply </p>\n\n<p>Yes it provided some nice boost. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "331968": "Can someone point me to a starter kernel for using Attention layers in RNN/CNN ? Any related online links to understand the concept will also be helpful.\n\nThanks \nShanth",
    "331972": "kernel: [attention-with-fasttext-embeddings][1]\n\nsome reading (altough it is hierarchical attention) [hierarchical-attention-networks.pdf][2]\n\n\n  [1]: https://www.kaggle.com/sanghan/attention-with-fasttext-embeddings\n  [2]: http://www.cs.cmu.edu/~./hovy/papers/16HLT-hierarchical-attention-networks.pdf",
    "331974": "Thank you so much",
    "331987": "I just added [this simple attention][1] after the RNN Layer . Beware of the training time ( if ever you don't have access to GPU)\n\nHierarchical  attention  did not help that much and increased futhermore the training time (compared to simple Atention) \n\n\n  [1]: https://gist.github.com/cbaziotis/6428df359af27d58078ca5ed9792bd6d#file-attention-py",
    "331990": "Thanks Seringe",
    "343401": "Hey Serigne, did attention provide a nice boost for you? Cheers.",
    "343969": "Hi den3b...\n\nSorry for the late reply \n\nYes it provided some nice boost."
  },
  "source": "meta"
}