In the models thread, sayit and Gilberto Titericz Junior mentioned using pseudo-labels for extra data. As far as I understand, you'd train a model, probably a neural network, on labeled data and then predict labels for unlabeled data. Then use the extra\r\n data with those predicted pseudo-labels for training. Why would it work?
", "messages": [ { "id": "24931", "postDate": "05/30/2013 10:31:33", "content": "In the models thread, sayit and Gilberto Titericz Junior mentioned using pseudo-labels for extra data. As far as I understand, you'd train a model, probably a neural network, on labeled data and then predict labels for unlabeled data. Then use the extra\r\n data with those predicted pseudo-labels for training. Why would it work?
", "rawMarkdown": "", "votes": null }, { "id": "24932", "postDate": "05/30/2013 11:05:04", "content": "See this paper entitled 'Entropy Regularization': http://www.iro.umontreal.ca/~lisa/publications2/index.php/publications/show/8
\r\nIt favors a low-density separation between classes, a commonly assumed prior for classification problems in machine learning.
\r\n", "rawMarkdown": "", "votes": null }, { "id": "24961", "postDate": "05/30/2013 18:35:50", "content": "The effect of supervised training unlabeled data with pseudo-labels is that the (neural) network outputs of unlabeled data are closer to 1 or 0 than training only labeled data.
\r\nI think that this is some kind of contractive regularization, I need to study more for detailed theoretical background (including the article above:))
\r\n", "rawMarkdown": "", "votes": null }, { "id": "24962", "postDate": "05/30/2013 19:29:41", "content": "\r\n[quote=sayit;24961]
\r\nThe effect of supervised training unlabeled data with pseudo-labels is that the (neural) network outputs of unlabeled data are closer to 1 or 0 than training only labeled data.
\r\nI think that this is some kind of contractive regularization, I need to study more for detailed theoretical background (including the article above:))
\r\n\r\n[/quote]
\r\nIntuitively, a network learns to confirm its own suspicions, so it makes sense.
\r\nAs far as I understand, the key concept here is a cluster assumption, meaning that each class forms a cluster and the clusters are separated by low density regions. The intro in this paper sums it up nicely:
\r\nChapelle and Zien: Semi-Supervised Classification by Low Density Separation
\r\nhttp://www.kyb.mpg.de/publications/pdfs/pdf2899.pdf
\r\n", "rawMarkdown": "", "votes": null }, { "id": "25016", "postDate": "06/01/2013 01:51:39", "content": "In my experiments, pseudo-labels are re-calculated every weights update in training with labeled and unlabeled data\r\nsimultaneously. If we calculate pseudo-label once after training with only labeled data, pseudo-label might be less accurate because the network is overfitted. After training several initial epochs with only\r\n labeled data, the network should be trained with labeled data and unlabeled data using continuously re-calculated pseudo-label. This scheme improve the generalization performance really.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 24932, "author_name": "yoshuabengio", "author_url": "", "post_date": "05/30/2013 11:05:04", "content": "See this paper entitled 'Entropy Regularization': http://www.iro.umontreal.ca/~lisa/publications2/index.php/publications/show/8
\r\nIt favors a low-density separation between classes, a commonly assumed prior for classification problems in machine learning.
\r\n", "votes": null, "replies": [] }, { "id": 24961, "author_name": "donghyun", "author_url": "", "post_date": "05/30/2013 18:35:50", "content": "The effect of supervised training unlabeled data with pseudo-labels is that the (neural) network outputs of unlabeled data are closer to 1 or 0 than training only labeled data.
\r\nI think that this is some kind of contractive regularization, I need to study more for detailed theoretical background (including the article above:))
\r\n", "votes": null, "replies": [] }, { "id": 24962, "author_name": "zygmunt", "author_url": "", "post_date": "05/30/2013 19:29:41", "content": "\r\n[quote=sayit;24961]
\r\nThe effect of supervised training unlabeled data with pseudo-labels is that the (neural) network outputs of unlabeled data are closer to 1 or 0 than training only labeled data.
\r\nI think that this is some kind of contractive regularization, I need to study more for detailed theoretical background (including the article above:))
\r\n\r\n[/quote]
\r\nIntuitively, a network learns to confirm its own suspicions, so it makes sense.
\r\nAs far as I understand, the key concept here is a cluster assumption, meaning that each class forms a cluster and the clusters are separated by low density regions. The intro in this paper sums it up nicely:
\r\nChapelle and Zien: Semi-Supervised Classification by Low Density Separation
\r\nhttp://www.kyb.mpg.de/publications/pdfs/pdf2899.pdf
\r\n", "votes": null, "replies": [] }, { "id": 25016, "author_name": "donghyun", "author_url": "", "post_date": "06/01/2013 01:51:39", "content": "In my experiments, pseudo-labels are re-calculated every weights update in training with labeled and unlabeled data\r\nsimultaneously. If we calculate pseudo-label once after training with only labeled data, pseudo-label might be less accurate because the network is overfitted. After training several initial epochs with only\r\n labeled data, the network should be trained with labeled data and unlabeled data using continuously re-calculated pseudo-label. This scheme improve the generalization performance really.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "24931": "", "24932": "", "24961": "", "24962": "", "25016": "" }, "source": "meta" }