{
  "id": 57570,
  "title": "Maybe you can try LDA...",
  "url": "/competitions/avito-demand-prediction/discussion/57570",
  "author_name": "",
  "post_date": "2018-05-25T12:52:19.414460300Z",
  "votes": 13,
  "comment_count": 17,
  "views": 0,
  "content": "<h2>What's LDA?</h2>\n\n<p><em><strong>Latent Dirichlet Allocation (LDA)</strong></em> is a generative statistical model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar. Here you can learn more about <a href=\"https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation\">LDA</a>.</p>\n\n<h2>What can LDA do?</h2>\n\n<p>In LDA, each document may be viewed as a mixture of various topics where each document is considered to have a set of topics that are assigned to it via LDA. So we can quickly understand the theme of an article through LDA topics.</p>\n\n<h2>How to use LDA?</h2>\n\n<p>In addition to directly extracting the topic of the article, just like word2vec, the matrix generated during LDA training is also very useful. We can use topic-document distribution matrix as a text feature for classification and other tasks.</p>\n\n<p>I tried to use LDA in the <code>Toxic Comment Classification</code>. But after some trial, there was no obvious improvement. But in Avito, I got a 0.0008 promotion on local-cv and lb by adding LDA feature. This result surprised me, because I think LDA is generally suitable for longer text. I think it might be for the following reasons：</p>\n\n<ul>\n<li>Compared to <code>Toxic Comment Classification</code>, the features I use are weaker now.</li>\n<li>LDA's performance in different languages will be different.</li>\n<li>...</li>\n</ul>\n\n<p>In fact, this is the first time that I have achieved significant improvement through the use of LDA. So I think if you have nothing to do, you can try LDA. Then we can talk about whether LDA is really effective :). At the same time, there are many similar algorithms, such as <code>pLSA</code> and <code>HDP</code>, which can also be tried.</p>",
  "messages": [
    {
      "id": "333582",
      "postDate": "05/25/2018 12:52:19",
      "content": "<h2>What's LDA?</h2>\n\n<p><em><strong>Latent Dirichlet Allocation (LDA)</strong></em> is a generative statistical model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar. Here you can learn more about <a href=\"https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation\">LDA</a>.</p>\n\n<h2>What can LDA do?</h2>\n\n<p>In LDA, each document may be viewed as a mixture of various topics where each document is considered to have a set of topics that are assigned to it via LDA. So we can quickly understand the theme of an article through LDA topics.</p>\n\n<h2>How to use LDA?</h2>\n\n<p>In addition to directly extracting the topic of the article, just like word2vec, the matrix generated during LDA training is also very useful. We can use topic-document distribution matrix as a text feature for classification and other tasks.</p>\n\n<p>I tried to use LDA in the <code>Toxic Comment Classification</code>. But after some trial, there was no obvious improvement. But in Avito, I got a 0.0008 promotion on local-cv and lb by adding LDA feature. This result surprised me, because I think LDA is generally suitable for longer text. I think it might be for the following reasons：</p>\n\n<ul>\n<li>Compared to <code>Toxic Comment Classification</code>, the features I use are weaker now.</li>\n<li>LDA's performance in different languages will be different.</li>\n<li>...</li>\n</ul>\n\n<p>In fact, this is the first time that I have achieved significant improvement through the use of LDA. So I think if you have nothing to do, you can try LDA. Then we can talk about whether LDA is really effective :). At the same time, there are many similar algorithms, such as <code>pLSA</code> and <code>HDP</code>, which can also be tried.</p>",
      "rawMarkdown": "## What's LDA?\n\n\n***Latent Dirichlet Allocation (LDA)*** is a generative statistical model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar. Here you can learn more about [LDA][1].\n\n## What can LDA do?\nIn LDA, each document may be viewed as a mixture of various topics where each document is considered to have a set of topics that are assigned to it via LDA. So we can quickly understand the theme of an article through LDA topics.\n\n\n## How to use LDA?\nIn addition to directly extracting the topic of the article, just like word2vec, the matrix generated during LDA training is also very useful. We can use topic-document distribution matrix as a text feature for classification and other tasks.\n\n\n\nI tried to use LDA in the `Toxic Comment Classification`. But after some trial, there was no obvious improvement. But in Avito, I got a 0.0008 promotion on local-cv and lb by adding LDA feature. This result surprised me, because I think LDA is generally suitable for longer text. I think it might be for the following reasons：\n\n* Compared to `Toxic Comment Classification`, the features I use are weaker now.\n* LDA's performance in different languages will be different.\n* ...\n\nIn fact, this is the first time that I have achieved significant improvement through the use of LDA. So I think if you have nothing to do, you can try LDA. Then we can talk about whether LDA is really effective :). At the same time, there are many similar algorithms, such as `pLSA` and `HDP`, which can also be tried.\n\n\n  [1]: https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation \"LDA\"",
      "votes": null
    },
    {
      "id": "333591",
      "postDate": "05/25/2018 13:15:55",
      "content": "<p>It looks great. Thanks. Something to try in this weekend. </p>",
      "rawMarkdown": "It looks great. Thanks. Something to try in this weekend.",
      "votes": null
    },
    {
      "id": "333606",
      "postDate": "05/25/2018 13:47:25",
      "content": "<p>Hi, nice to see you again.   You lost a gold medal in Toxic. Hope this time you can get one gold medal.  Good luck!</p>",
      "rawMarkdown": "Hi, nice to see you again.   You lost a gold medal in Toxic. Hope this time you can get one gold medal.  Good luck!",
      "votes": null
    },
    {
      "id": "333609",
      "postDate": "05/25/2018 13:51:11",
      "content": "<p>I wouldn't say loose... Rather cost your team members... But hey, maybe he (and some others) learned something.</p>",
      "rawMarkdown": "I wouldn't say loose... Rather cost your team members... But hey, maybe he (and some others) learned something.",
      "votes": null
    },
    {
      "id": "333908",
      "postDate": "05/26/2018 02:37:27",
      "content": "<p>Pls forgive me poor english. : (   I mean he is a talent guy and hope he can fly again in kaggle after the toxic lesson.    BTW,  hi Dieter,  I like the kernels  which you shared.  In China, I never see  people over 60 years old can do these like you in kaggle.   You are my idol.   Hope you can get the first gold medal in Avito.  : )</p>",
      "rawMarkdown": "Pls forgive me poor english. : (   I mean he is a talent guy and hope he can fly again in kaggle after the toxic lesson.    BTW,  hi Dieter,  I like the kernels  which you shared.  In China, I never see  people over 60 years old can do these like you in kaggle.   You are my idol.   Hope you can get the first gold medal in Avito.  : )",
      "votes": null
    },
    {
      "id": "333930",
      "postDate": "05/26/2018 04:01:42",
      "content": "<p>I tried LDA, since one of the top solutions of another challenge used those, but for me it did not help.</p>",
      "rawMarkdown": "I tried LDA, since one of the top solutions of another challenge used those, but for me it did not help.",
      "votes": null
    },
    {
      "id": "334323",
      "postDate": "05/27/2018 00:44:15",
      "content": "<p>May I know the number of topics you used?</p>",
      "rawMarkdown": "May I know the number of topics you used?",
      "votes": null
    },
    {
      "id": "334374",
      "postDate": "05/27/2018 06:51:12",
      "content": "<p>5</p>",
      "rawMarkdown": "5",
      "votes": null
    },
    {
      "id": "334388",
      "postDate": "05/27/2018 07:31:01",
      "content": "<p>5 seems too small...</p>",
      "rawMarkdown": "5 seems too small...",
      "votes": null
    },
    {
      "id": "334463",
      "postDate": "05/27/2018 12:54:57",
      "content": "<p>I tried 50 and didn't improve either.</p>",
      "rawMarkdown": "I tried 50 and didn't improve either.",
      "votes": null
    },
    {
      "id": "334478",
      "postDate": "05/27/2018 14:41:15",
      "content": "<p>probabaly you already have good features </p>",
      "rawMarkdown": "probabaly you already have good features",
      "votes": null
    },
    {
      "id": "334485",
      "postDate": "05/27/2018 15:06:07",
      "content": "<p>could be (my lgb is at 0.2207 LB). None of the unsupervised methods (SVD, LDA, libfm) helped improving</p>",
      "rawMarkdown": "could be (my lgb is at 0.2207 LB). None of the unsupervised methods (SVD, LDA, libfm) helped improving",
      "votes": null
    },
    {
      "id": "334489",
      "postDate": "05/27/2018 15:12:42",
      "content": "<p>unsupervised methods to me it's more like a chance to learn new things rather improve scores </p>",
      "rawMarkdown": "unsupervised methods to me it's more like a chance to learn new things rather improve scores",
      "votes": null
    },
    {
      "id": "335104",
      "postDate": "05/29/2018 07:06:55",
      "content": "<p>In China, people older than 35 have to do management, because boss thinks you are too old to write code and to work overtime.</p>",
      "rawMarkdown": "In China, people older than 35 have to do management, because boss thinks you are too old to write code and to work overtime.",
      "votes": null
    },
    {
      "id": "335120",
      "postDate": "05/29/2018 07:51:09",
      "content": "<p>LatentDirichletAllocation is very slow?I tried on kaggle kernel,it takes too much time by 30 jobs,I can't get any result.Is there any way to speed up?</p>",
      "rawMarkdown": "LatentDirichletAllocation is very slow?I tried on kaggle kernel,it takes too much time by 30 jobs,I can't get any result.Is there any way to speed up?",
      "votes": null
    },
    {
      "id": "335142",
      "postDate": "05/29/2018 08:56:27",
      "content": "<p>Are you using online learning algo? </p>",
      "rawMarkdown": "Are you using online learning algo?",
      "votes": null
    },
    {
      "id": "335203",
      "postDate": "05/29/2018 11:14:47",
      "content": "<p>I have make it out,it takes one hour to handle title with 30 threads,But I dont know how to use this topic，sad</p>",
      "rawMarkdown": "I have make it out,it takes one hour to handle title with 30 threads,But I dont know how to use this topic，sad",
      "votes": null
    },
    {
      "id": "335301",
      "postDate": "05/29/2018 14:45:08",
      "content": "<p>You may try to use topics as feature for supervised learning</p>",
      "rawMarkdown": "You may try to use topics as feature for supervised learning",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 333591,
      "author_name": "yimacs",
      "author_url": "",
      "post_date": "05/25/2018 13:15:55",
      "content": "<p>It looks great. Thanks. Something to try in this weekend. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 333606,
      "author_name": "qinhui1999",
      "author_url": "",
      "post_date": "05/25/2018 13:47:25",
      "content": "<p>Hi, nice to see you again.   You lost a gold medal in Toxic. Hope this time you can get one gold medal.  Good luck!</p>",
      "votes": null,
      "replies": [
        {
          "id": 333609,
          "author_name": "christofhenkel",
          "author_url": "",
          "post_date": "05/25/2018 13:51:11",
          "content": "<p>I wouldn't say loose... Rather cost your team members... But hey, maybe he (and some others) learned something.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 333908,
          "author_name": "qinhui1999",
          "author_url": "",
          "post_date": "05/26/2018 02:37:27",
          "content": "<p>Pls forgive me poor english. : (   I mean he is a talent guy and hope he can fly again in kaggle after the toxic lesson.    BTW,  hi Dieter,  I like the kernels  which you shared.  In China, I never see  people over 60 years old can do these like you in kaggle.   You are my idol.   Hope you can get the first gold medal in Avito.  : )</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 335104,
          "author_name": "jinmahkust",
          "author_url": "",
          "post_date": "05/29/2018 07:06:55",
          "content": "<p>In China, people older than 35 have to do management, because boss thinks you are too old to write code and to work overtime.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 333930,
      "author_name": "christofhenkel",
      "author_url": "",
      "post_date": "05/26/2018 04:01:42",
      "content": "<p>I tried LDA, since one of the top solutions of another challenge used those, but for me it did not help.</p>",
      "votes": null,
      "replies": [
        {
          "id": 334323,
          "author_name": "huangkh19951228",
          "author_url": "",
          "post_date": "05/27/2018 00:44:15",
          "content": "<p>May I know the number of topics you used?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 334374,
          "author_name": "christofhenkel",
          "author_url": "",
          "post_date": "05/27/2018 06:51:12",
          "content": "<p>5</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 334388,
          "author_name": "yimacs",
          "author_url": "",
          "post_date": "05/27/2018 07:31:01",
          "content": "<p>5 seems too small...</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 334463,
          "author_name": "huangkh19951228",
          "author_url": "",
          "post_date": "05/27/2018 12:54:57",
          "content": "<p>I tried 50 and didn't improve either.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 334478,
          "author_name": "yimacs",
          "author_url": "",
          "post_date": "05/27/2018 14:41:15",
          "content": "<p>probabaly you already have good features </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 334485,
          "author_name": "christofhenkel",
          "author_url": "",
          "post_date": "05/27/2018 15:06:07",
          "content": "<p>could be (my lgb is at 0.2207 LB). None of the unsupervised methods (SVD, LDA, libfm) helped improving</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 334489,
          "author_name": "yimacs",
          "author_url": "",
          "post_date": "05/27/2018 15:12:42",
          "content": "<p>unsupervised methods to me it's more like a chance to learn new things rather improve scores </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 335120,
      "author_name": "liuhdsgoal",
      "author_url": "",
      "post_date": "05/29/2018 07:51:09",
      "content": "<p>LatentDirichletAllocation is very slow?I tried on kaggle kernel,it takes too much time by 30 jobs,I can't get any result.Is there any way to speed up?</p>",
      "votes": null,
      "replies": [
        {
          "id": 335142,
          "author_name": "yimacs",
          "author_url": "",
          "post_date": "05/29/2018 08:56:27",
          "content": "<p>Are you using online learning algo? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 335203,
          "author_name": "liuhdsgoal",
          "author_url": "",
          "post_date": "05/29/2018 11:14:47",
          "content": "<p>I have make it out,it takes one hour to handle title with 30 threads,But I dont know how to use this topic，sad</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 335301,
          "author_name": "insaff",
          "author_url": "",
          "post_date": "05/29/2018 14:45:08",
          "content": "<p>You may try to use topics as feature for supervised learning</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "333582": "## What's LDA?\n\n\n***Latent Dirichlet Allocation (LDA)*** is a generative statistical model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar. Here you can learn more about [LDA][1].\n\n## What can LDA do?\nIn LDA, each document may be viewed as a mixture of various topics where each document is considered to have a set of topics that are assigned to it via LDA. So we can quickly understand the theme of an article through LDA topics.\n\n\n## How to use LDA?\nIn addition to directly extracting the topic of the article, just like word2vec, the matrix generated during LDA training is also very useful. We can use topic-document distribution matrix as a text feature for classification and other tasks.\n\n\n\nI tried to use LDA in the `Toxic Comment Classification`. But after some trial, there was no obvious improvement. But in Avito, I got a 0.0008 promotion on local-cv and lb by adding LDA feature. This result surprised me, because I think LDA is generally suitable for longer text. I think it might be for the following reasons：\n\n* Compared to `Toxic Comment Classification`, the features I use are weaker now.\n* LDA's performance in different languages will be different.\n* ...\n\nIn fact, this is the first time that I have achieved significant improvement through the use of LDA. So I think if you have nothing to do, you can try LDA. Then we can talk about whether LDA is really effective :). At the same time, there are many similar algorithms, such as `pLSA` and `HDP`, which can also be tried.\n\n\n  [1]: https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation \"LDA\"",
    "333591": "It looks great. Thanks. Something to try in this weekend.",
    "333606": "Hi, nice to see you again.   You lost a gold medal in Toxic. Hope this time you can get one gold medal.  Good luck!",
    "333609": "I wouldn't say loose... Rather cost your team members... But hey, maybe he (and some others) learned something.",
    "333908": "Pls forgive me poor english. : (   I mean he is a talent guy and hope he can fly again in kaggle after the toxic lesson.    BTW,  hi Dieter,  I like the kernels  which you shared.  In China, I never see  people over 60 years old can do these like you in kaggle.   You are my idol.   Hope you can get the first gold medal in Avito.  : )",
    "333930": "I tried LDA, since one of the top solutions of another challenge used those, but for me it did not help.",
    "334323": "May I know the number of topics you used?",
    "334374": "5",
    "334388": "5 seems too small...",
    "334463": "I tried 50 and didn't improve either.",
    "334478": "probabaly you already have good features",
    "334485": "could be (my lgb is at 0.2207 LB). None of the unsupervised methods (SVD, LDA, libfm) helped improving",
    "334489": "unsupervised methods to me it's more like a chance to learn new things rather improve scores",
    "335104": "In China, people older than 35 have to do management, because boss thinks you are too old to write code and to work overtime.",
    "335120": "LatentDirichletAllocation is very slow?I tried on kaggle kernel,it takes too much time by 30 jobs,I can't get any result.Is there any way to speed up?",
    "335142": "Are you using online learning algo?",
    "335203": "I have make it out,it takes one hour to handle title with 30 threads,But I dont know how to use this topic，sad",
    "335301": "You may try to use topics as feature for supervised learning"
  },
  "source": "meta"
}