{ "id": 4302, "title": "Methods and Algorithms", "url": "/competitions/challenges-in-representation-learning-the-black-box-learning-challenge/discussion/4302", "author_name": "", "post_date": "2013-04-14T00:36:30.353Z", "votes": null, "comment_count": 35, "views": 9385, "content": "

Just for fun I thought I'd see if anybody wants to talk about what methods and algorithms they've tried (or are working on).  GIven the moniker of the early leader (i.e. RBM) I think the early leader's strategy is clear - and I may resurrect my own Restricted\r\n Boltzmann Machine implementation (circa 2008/2009) and give it a shot.

\r\n

But personally I wanted to try simpler things first - just to add to the "benchmark" collection.

\r\n

My first attempt was a plain vanilla multiclass random forest, followed by a 9-way attempt (making each feature binary).  The latter scored higher: 0.36540, better than I thought it would.

\r\n

So far I haven't attempted to do anything (unsupervised or otherwise) with the unlabeled data, and I'm not sure I will.  Depends on how much time I can scrape together for this.

", "messages": [ { "id": "22757", "postDate": "04/14/2013 00:36:30", "content": "

Just for fun I thought I'd see if anybody wants to talk about what methods and algorithms they've tried (or are working on).  GIven the moniker of the early leader (i.e. RBM) I think the early leader's strategy is clear - and I may resurrect my own Restricted\r\n Boltzmann Machine implementation (circa 2008/2009) and give it a shot.

\r\n

But personally I wanted to try simpler things first - just to add to the "benchmark" collection.

\r\n

My first attempt was a plain vanilla multiclass random forest, followed by a 9-way attempt (making each feature binary).  The latter scored higher: 0.36540, better than I thought it would.

\r\n

So far I haven't attempted to do anything (unsupervised or otherwise) with the unlabeled data, and I'm not sure I will.  Depends on how much time I can scrape together for this.

", "rawMarkdown": "", "votes": null }, { "id": "22758", "postDate": "04/14/2013 00:46:09", "content": "

Since I'm an organizer of the challenge, I won't be giving any opinions on what algorithm to use. But I think it's OK for me to point out that the baseline code provided for this challenge can be modified pretty easily to include RBMs if they're what you\r\n want to try.

\r\n

The baseline code for this challenge is here:

\r\n

http://github.com/lisa-lab/pylearn2/tree/master/pylearn2/scripts/icml_2013_wrepl/black_box

\r\n

The pylearn2 RBM is here:

\r\n

http://github.com/lisa-lab/pylearn2/blob/master/pylearn2/models/rbm.py

\r\n

It should be reasonably easy to modify the baseline demo script to incorporate RBMs, using the RBM_Layer class of the MLP:

\r\n

http://github.com/lisa-lab/pylearn2/blob/master/pylearn2/models/mlp.py

", "rawMarkdown": "", "votes": null }, { "id": "22760", "postDate": "04/14/2013 00:52:42", "content": "

[quote=Ian Goodfellow;22758]

\r\n

...I think it's OK for me to point out that the baseline code provided for this challenge can be modified pretty easily to include RBMs if they're what you want to try...

\r\n

[/quote]

\r\n

Very true, but I have a "thing" about using as much of my own code as possible.  That way nothing is really a black box.  Just a personal quirk, though one that tends to fall by the wayside when it gets to crunch time.  Pragmatism often trumps philosophy.

", "rawMarkdown": "", "votes": null }, { "id": "22801", "postDate": "04/14/2013 19:09:26", "content": "

I've put in a few submissions continuing my "play with vw" exploration... Using quadratic features gets me up into the range of the top three benchmarks. Whoo!

", "rawMarkdown": "", "votes": null }, { "id": "22869", "postDate": "04/15/2013 23:28:49", "content": "

[quote=Aaron Schumacher;22801]

\r\n

I've put in a few submissions continuing my "play with vw" exploration... Using quadratic features gets me up into the range of the top three benchmarks. Whoo!

\r\n

[/quote]

\r\n

Nice!  I don't know much about vw (yet - it's on my list of things to learn), but from what I can tell it doesn't have any semi-supervised learning methods built in.  So were you using just the labeled data?

\r\n

I thought I'd try some old-school semi-supervised methods next - gaussian mixture stuff most likely - just to see if there's any clearly discernable value in the unlabeled data.  Memories of a prior (and similar) Kaggle competition (https://www.kaggle.com/c/SemiSupervisedFeatureLearning),\r\n where the winner didn't use the unlabeled data at all, have made me a bit gun-shy.

", "rawMarkdown": "", "votes": null }, { "id": "22874", "postDate": "04/16/2013 01:18:44", "content": "

This challenge has 1000 labeled examples, covering 9 classes which are not balanced either. This makes getting 0.99 AUC (or 99% accuracy) significantly more difficult (the underlying problem is also not particularly trivial, which adds to the complexity\r\n of course).

\r\n

Nonetheless, part of the point of this workshop is to add yet another data point to the debate on whether unsupervised learning is indeed helpful. A\r\nhypothesis is that it is helpful, and this competition will provide data to support it (or not).

\r\n

Dumitru

", "rawMarkdown": "", "votes": null }, { "id": "22897", "postDate": "04/16/2013 14:42:59", "content": "

@YetiMan, yes, I don't know of any unsupervised/semi-supervised stuff in vw, so I was just using the labeled data. I'm not sure yet what I want to do with the unlabeled data. I thought about trying to label it based on my label-trained model and then train\r\n further using those guesses, but I don't know that that would be helpful. Thanks for the reference to the old comp! Interesting...

", "rawMarkdown": "", "votes": null }, { "id": "22978", "postDate": "04/17/2013 15:24:12", "content": "

[quote=Aaron Schumacher;22897]

\r\n

@YetiMan, yes, I don't know of any unsupervised/semi-supervised stuff in vw, so I was just using the labeled data. I'm not sure yet what I want to do with the unlabeled data. I thought about trying to label it based on my label-trained model and then train\r\n further using those guesses, but I don't know that that would be helpful. Thanks for the reference to the old comp! Interesting...

\r\n

[/quote]

\r\n

I've used vw quite a bit, and i don't think they have unsupervised stuff. But o do know that this one has: https://code.google.com/p/sofia-ml/wiki/SofiaKMeans

", "rawMarkdown": "", "votes": null }, { "id": "22983", "postDate": "04/17/2013 16:21:24", "content": "

Neat! Yeah, I wasn't suggesting that vw could do anything unsupervised, I was suggesting the probably bad idea of using a label-trained model to label the unlabeled data and then training more off of that. Using a real unsupervised technique is likely to\r\n be better, I imagine. :)

", "rawMarkdown": "", "votes": null }, { "id": "22985", "postDate": "04/17/2013 17:00:13", "content": "

[quote=Aaron Schumacher;22983]

\r\n

Neat! Yeah, I wasn't suggesting that vw could do anything unsupervised, I was suggesting the probably bad idea of using a label-trained model to label the unlabeled data and then training more off of that. Using a real unsupervised technique is likely to\r\n be better, I imagine. :)

\r\n

[/quote]

\r\n

I tried that a couple of days ago, with labels imputed via gradient boosted decision trees (chosen only because I had the code handy).  As you surmised, the results were horrible.

\r\n

To define "horrible" more concretely:

\r\n\r\n

Last night's attempt at semi-supervised learning (via TSVM) was also an abysmal failure.  Much worse than either GBDT or "random forest" on only training data.  To be fair, though, I only used 20% of the unlabeled data in order to decrease training time. \r\n So I haven't completely given up on using the unlabeled data, but thus far it's looking more harmful than helpful - which could simply be a sign that I'm choosing poor methods (or that there are bugs in my code, or that there's too much "noise" in the data\r\n overall, or...).

\r\n

Unfortunately the time I have available to spend on this is very limited, so I need to decide which path to take: supervised using only labeled data, semi-supervised, or unsupervised.

", "rawMarkdown": "", "votes": null }, { "id": "23238", "postDate": "04/20/2013 19:20:45", "content": "

[quote=Aaron Schumacher;22801]

\r\n

I've put in a few submissions continuing my "play with vw" exploration... Using quadratic features gets me up into the range of the top three benchmarks. Whoo!

\r\n

[/quote]

\r\n

By the way, what is vw?

\r\n

I tried to google it, and only results were related to volkswagen.

", "rawMarkdown": "", "votes": null }, { "id": "23239", "postDate": "04/20/2013 19:22:03", "content": "

Vowpal Wabbit

", "rawMarkdown": "", "votes": null }, { "id": "23561", "postDate": "04/28/2013 01:40:44", "content": "

I've been wondering for a while whether the test set has the same distribution of classes as the training set.  I've just submitted a probe full of 10000 "8.0"'s -- the result was 0.06520 which is pretty close to the 0.073 of the training set.

", "rawMarkdown": "", "votes": null }, { "id": "23624", "postDate": "04/29/2013 14:58:23", "content": "

Yop,

\r\n

I tried sciki-learn's \r\nRandom Forests w. n_estimators=100 and scored 0.34 using just the labelled data.

\r\n

GBDT, SVM, kNN, SGD, etc did perform horribly.

\r\n

I tried to learn it as a regression problem (as the class names might imply), but with no luck either using Random Forest Regressor.

\r\n

I also tried the \r\nLabelPropagation / LabelSpreading implementations in sklearn to make use of unsupervised data but was quite unlucky so far (

\r\n

I'm wondering if this problem was not utterly designed to be tackled with (modern) neural networks (aka deep learning)  stuff...

", "rawMarkdown": "", "votes": null }, { "id": "23630", "postDate": "04/29/2013 16:04:25", "content": "

The problem was designed to be difficult, and we chose the amount of labeled and unlabeled data to give an advantage to algorithms that can run in semi-supervised mode. Beyond that we were not trying to make it work well with any particular kind of learning\r\n algorithm. I think the problem you're observing is that the learning algorithms you're trying to run only work well if they have good features as input, but here you need to learn the features.

", "rawMarkdown": "", "votes": null }, { "id": "23635", "postDate": "04/29/2013 16:30:17", "content": "

I am new to data crunching and meanwhile in the process of shifting from R to Python. I am an ordinary R user.

\r\n

May I ask you guys a quick question that how much time does it take for you to get a result on your computer, given that algorithms and codes have been written? Does it take several hours running? Thank you!

", "rawMarkdown": "", "votes": null }, { "id": "23638", "postDate": "04/29/2013 16:38:37", "content": "

Mmm... what i was trying to say was that deep learning seems a paradigm of choice for this problem given its popularity in the recent years. Or so it seems from my limited knowledge of the semi supervised field. 

\r\n

 

\r\n

Of course if you or anyone else has pointers to other ss approaches that might work, i'm all ears ;-)

", "rawMarkdown": "", "votes": null }, { "id": "23639", "postDate": "04/29/2013 16:41:09", "content": "

Since I'm organizing the contest I probably shouldn't provide advice for how to compete in it, beyond helping people troubleshoot pylearn2 stuff.

", "rawMarkdown": "", "votes": null }, { "id": "23645", "postDate": "04/29/2013 17:00:06", "content": "

Of course that wasn't what I was asking !!!

\r\n

It's just that I was seeking new fields to explore and ideas ...

\r\n

if s.o. tells me I could look at the, say, manifold learning or co-training or self-training or ... he's not necessarily telling me how to solve the problem but rather educating me about existing, but possibly totally irrelevant approaches out there.\r\n

", "rawMarkdown": "", "votes": null }, { "id": "23683", "postDate": "04/30/2013 04:32:10", "content": "

is there any example on how to use Restricted boltzmann machines (rbm.py) with a sample .yaml file

", "rawMarkdown": "", "votes": null }, { "id": "23684", "postDate": "04/30/2013 04:35:19", "content": "

pylearn2/scripts/tutorials/grbm_smd and pylearn2/scripts/tutorials/deep_trainer both use RBMs.

", "rawMarkdown": "", "votes": null }, { "id": "23687", "postDate": "04/30/2013 07:44:07", "content": "

that is for a binary RBM

\r\n

Can you let us know which function in rbm.py is a multi-class rbm?

", "rawMarkdown": "", "votes": null }, { "id": "23698", "postDate": "04/30/2013 15:23:50", "content": "

Well, the hidden units are binary, but the visible units are real-valued.

\r\n

I don't think that rbm.py supports softmax variables, but dbm.py does. See pylearn2/scripts/tutorials/dbm_demo for a basic example of the interface.

", "rawMarkdown": "", "votes": null }, { "id": "23709", "postDate": "04/30/2013 16:19:39", "content": "

Thanks Ian.

\r\n

I guess the "y" layer must be softmax.

\r\n

rbm.yaml in the dbm_demo tutorial has no mention of the y interface

", "rawMarkdown": "", "votes": null }, { "id": "23711", "postDate": "04/30/2013 16:23:56", "content": "

pylearn2.models.dbm.Softmax. There's not a nice flashy demo script for every feature of the library. If someone would like to make a pull request with more demos, that'd be great.

", "rawMarkdown": "", "votes": null }, { "id": "23712", "postDate": "04/30/2013 16:26:57", "content": "

ok, difficult for seasoned R programmers!

\r\n

visible_layer: !obj:pylearn2.models.dbm.BinaryVector

\r\n

to be replaced by

\r\n

visible_layer: !obj:pylearn2.models.dbm.Softmax

\r\n

 

\r\n

Correct?

", "rawMarkdown": "", "votes": null }, { "id": "23713", "postDate": "04/30/2013 16:30:39", "content": "

Set the cost's supervised flag to True and use Softmax as the last hidden layer.

\r\n

The DBM code generally doesn't consider the class labels as "visible" because they're unobserved for test set data.

\r\n

The visible_layer for this challenge should be real-valued. I'm not sure if there's a real-valued DBM layer in the library right now.

", "rawMarkdown": "", "votes": null }, { "id": "23715", "postDate": "04/30/2013 17:03:33", "content": "

https://github.com/lisa-lab/pylearn2/tree/master/pylearn2/models/dbm

\r\n

There is no softmax here on github

", "rawMarkdown": "", "votes": null }, { "id": "23716", "postDate": "04/30/2013 17:16:10", "content": "

There is no softmax function in dbm.py also.

\r\n

It is only in mlp.py - Using same syntax gives following error:

\r\n

TypeError: Softmax does not support the following keywords: init_bias_target_marginals. Did you mean irange?

\r\n

!obj:pylearn2.models.dbm.Softmax {
\r\n                layer_name: 'h1',
\r\n                init_bias_target_marginals: *raw_train,
\r\n                # Initialize the weights to all 0s
\r\n                irange: .0,
\r\n                n_classes: 9
\r\n            }

\r\n


\r\n
\r\n

", "rawMarkdown": "", "votes": null }, { "id": "23717", "postDate": "04/30/2013 17:17:48", "content": "

https://github.com/lisa-lab/pylearn2/blob/master/pylearn2/models/dbm/__init__.py line 1612.

", "rawMarkdown": "", "votes": null }, { "id": "23920", "postDate": "05/05/2013 18:49:17", "content": "

My current algorithm (using python):

\r\n\r\n

Finally got 0.31 accuracy

", "rawMarkdown": "", "votes": null }, { "id": "23929", "postDate": "05/06/2013 04:17:22", "content": "\r\n

0.35 accuracy

", "rawMarkdown": "", "votes": null }, { "id": "23937", "postDate": "05/06/2013 12:16:35", "content": "

Just using Matlab NN toolbox without any pre processing. 

\r\n\r\n

0.40 accuracy

", "rawMarkdown": "", "votes": null }, { "id": "23988", "postDate": "05/07/2013 18:00:46", "content": "

[quote=Ulrich;23937]

\r\n

Just using Matlab NN toolbox without any pre processing. 

\r\n\r\n

0.40 accuracy

\r\n

[/quote]

\r\n

I run the Matlab NN toolbox and for more than 100 neurons I have 'out of memory' problem on 4GB machine. Did you face any problem as such?

", "rawMarkdown": "", "votes": null }, { "id": "23994", "postDate": "05/07/2013 19:15:50", "content": "

Hi Rafael,

\r\n

 Try using net.trainFcn = 'traingdx' as the matlab train function instead of the default 'trainlm' and you can have more than 100 neurons... also try to use a small initial learning rate if you use 2 hidden layers...

", "rawMarkdown": "", "votes": null }, { "id": "24013", "postDate": "05/08/2013 14:01:55", "content": "

Also     net.efficiency.memoryReduction     parameter may help.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 22758, "author_name": "iangoodfellow", "author_url": "", "post_date": "04/14/2013 00:46:09", "content": "

Since I'm an organizer of the challenge, I won't be giving any opinions on what algorithm to use. But I think it's OK for me to point out that the baseline code provided for this challenge can be modified pretty easily to include RBMs if they're what you\r\n want to try.

\r\n

The baseline code for this challenge is here:

\r\n

http://github.com/lisa-lab/pylearn2/tree/master/pylearn2/scripts/icml_2013_wrepl/black_box

\r\n

The pylearn2 RBM is here:

\r\n

http://github.com/lisa-lab/pylearn2/blob/master/pylearn2/models/rbm.py

\r\n

It should be reasonably easy to modify the baseline demo script to incorporate RBMs, using the RBM_Layer class of the MLP:

\r\n

http://github.com/lisa-lab/pylearn2/blob/master/pylearn2/models/mlp.py

", "votes": null, "replies": [] }, { "id": 22760, "author_name": "yetiman", "author_url": "", "post_date": "04/14/2013 00:52:42", "content": "

[quote=Ian Goodfellow;22758]

\r\n

...I think it's OK for me to point out that the baseline code provided for this challenge can be modified pretty easily to include RBMs if they're what you want to try...

\r\n

[/quote]

\r\n

Very true, but I have a "thing" about using as much of my own code as possible.  That way nothing is really a black box.  Just a personal quirk, though one that tends to fall by the wayside when it gets to crunch time.  Pragmatism often trumps philosophy.

", "votes": null, "replies": [] }, { "id": 22801, "author_name": "ajschumacher", "author_url": "", "post_date": "04/14/2013 19:09:26", "content": "

I've put in a few submissions continuing my "play with vw" exploration... Using quadratic features gets me up into the range of the top three benchmarks. Whoo!

", "votes": null, "replies": [] }, { "id": 22869, "author_name": "yetiman", "author_url": "", "post_date": "04/15/2013 23:28:49", "content": "

[quote=Aaron Schumacher;22801]

\r\n

I've put in a few submissions continuing my "play with vw" exploration... Using quadratic features gets me up into the range of the top three benchmarks. Whoo!

\r\n

[/quote]

\r\n

Nice!  I don't know much about vw (yet - it's on my list of things to learn), but from what I can tell it doesn't have any semi-supervised learning methods built in.  So were you using just the labeled data?

\r\n

I thought I'd try some old-school semi-supervised methods next - gaussian mixture stuff most likely - just to see if there's any clearly discernable value in the unlabeled data.  Memories of a prior (and similar) Kaggle competition (https://www.kaggle.com/c/SemiSupervisedFeatureLearning),\r\n where the winner didn't use the unlabeled data at all, have made me a bit gun-shy.

", "votes": null, "replies": [] }, { "id": 22874, "author_name": "dumitru0", "author_url": "", "post_date": "04/16/2013 01:18:44", "content": "

This challenge has 1000 labeled examples, covering 9 classes which are not balanced either. This makes getting 0.99 AUC (or 99% accuracy) significantly more difficult (the underlying problem is also not particularly trivial, which adds to the complexity\r\n of course).

\r\n

Nonetheless, part of the point of this workshop is to add yet another data point to the debate on whether unsupervised learning is indeed helpful. A\r\nhypothesis is that it is helpful, and this competition will provide data to support it (or not).

\r\n

Dumitru

", "votes": null, "replies": [] }, { "id": 22897, "author_name": "ajschumacher", "author_url": "", "post_date": "04/16/2013 14:42:59", "content": "

@YetiMan, yes, I don't know of any unsupervised/semi-supervised stuff in vw, so I was just using the labeled data. I'm not sure yet what I want to do with the unlabeled data. I thought about trying to label it based on my label-trained model and then train\r\n further using those guesses, but I don't know that that would be helpful. Thanks for the reference to the old comp! Interesting...

", "votes": null, "replies": [] }, { "id": 22978, "author_name": "leustagos", "author_url": "", "post_date": "04/17/2013 15:24:12", "content": "

[quote=Aaron Schumacher;22897]

\r\n

@YetiMan, yes, I don't know of any unsupervised/semi-supervised stuff in vw, so I was just using the labeled data. I'm not sure yet what I want to do with the unlabeled data. I thought about trying to label it based on my label-trained model and then train\r\n further using those guesses, but I don't know that that would be helpful. Thanks for the reference to the old comp! Interesting...

\r\n

[/quote]

\r\n

I've used vw quite a bit, and i don't think they have unsupervised stuff. But o do know that this one has: https://code.google.com/p/sofia-ml/wiki/SofiaKMeans

", "votes": null, "replies": [] }, { "id": 22983, "author_name": "ajschumacher", "author_url": "", "post_date": "04/17/2013 16:21:24", "content": "

Neat! Yeah, I wasn't suggesting that vw could do anything unsupervised, I was suggesting the probably bad idea of using a label-trained model to label the unlabeled data and then training more off of that. Using a real unsupervised technique is likely to\r\n be better, I imagine. :)

", "votes": null, "replies": [] }, { "id": 22985, "author_name": "yetiman", "author_url": "", "post_date": "04/17/2013 17:00:13", "content": "

[quote=Aaron Schumacher;22983]

\r\n

Neat! Yeah, I wasn't suggesting that vw could do anything unsupervised, I was suggesting the probably bad idea of using a label-trained model to label the unlabeled data and then training more off of that. Using a real unsupervised technique is likely to\r\n be better, I imagine. :)

\r\n

[/quote]

\r\n

I tried that a couple of days ago, with labels imputed via gradient boosted decision trees (chosen only because I had the code handy).  As you surmised, the results were horrible.

\r\n

To define "horrible" more concretely:

\r\n\r\n

Last night's attempt at semi-supervised learning (via TSVM) was also an abysmal failure.  Much worse than either GBDT or "random forest" on only training data.  To be fair, though, I only used 20% of the unlabeled data in order to decrease training time. \r\n So I haven't completely given up on using the unlabeled data, but thus far it's looking more harmful than helpful - which could simply be a sign that I'm choosing poor methods (or that there are bugs in my code, or that there's too much "noise" in the data\r\n overall, or...).

\r\n

Unfortunately the time I have available to spend on this is very limited, so I need to decide which path to take: supervised using only labeled data, semi-supervised, or unsupervised.

", "votes": null, "replies": [] }, { "id": 23238, "author_name": "benoitplante", "author_url": "", "post_date": "04/20/2013 19:20:45", "content": "

[quote=Aaron Schumacher;22801]

\r\n

I've put in a few submissions continuing my "play with vw" exploration... Using quadratic features gets me up into the range of the top three benchmarks. Whoo!

\r\n

[/quote]

\r\n

By the way, what is vw?

\r\n

I tried to google it, and only results were related to volkswagen.

", "votes": null, "replies": [] }, { "id": 23239, "author_name": "dumitru0", "author_url": "", "post_date": "04/20/2013 19:22:03", "content": "

Vowpal Wabbit

", "votes": null, "replies": [] }, { "id": 23561, "author_name": "johnclarke", "author_url": "", "post_date": "04/28/2013 01:40:44", "content": "

I've been wondering for a while whether the test set has the same distribution of classes as the training set.  I've just submitted a probe full of 10000 "8.0"'s -- the result was 0.06520 which is pretty close to the 0.073 of the training set.

", "votes": null, "replies": [] }, { "id": 23624, "author_name": "eustache", "author_url": "", "post_date": "04/29/2013 14:58:23", "content": "

Yop,

\r\n

I tried sciki-learn's \r\nRandom Forests w. n_estimators=100 and scored 0.34 using just the labelled data.

\r\n

GBDT, SVM, kNN, SGD, etc did perform horribly.

\r\n

I tried to learn it as a regression problem (as the class names might imply), but with no luck either using Random Forest Regressor.

\r\n

I also tried the \r\nLabelPropagation / LabelSpreading implementations in sklearn to make use of unsupervised data but was quite unlucky so far (

\r\n

I'm wondering if this problem was not utterly designed to be tackled with (modern) neural networks (aka deep learning)  stuff...

", "votes": null, "replies": [] }, { "id": 23630, "author_name": "iangoodfellow", "author_url": "", "post_date": "04/29/2013 16:04:25", "content": "

The problem was designed to be difficult, and we chose the amount of labeled and unlabeled data to give an advantage to algorithms that can run in semi-supervised mode. Beyond that we were not trying to make it work well with any particular kind of learning\r\n algorithm. I think the problem you're observing is that the learning algorithms you're trying to run only work well if they have good features as input, but here you need to learn the features.

", "votes": null, "replies": [] }, { "id": 23635, "author_name": "weiden", "author_url": "", "post_date": "04/29/2013 16:30:17", "content": "

I am new to data crunching and meanwhile in the process of shifting from R to Python. I am an ordinary R user.

\r\n

May I ask you guys a quick question that how much time does it take for you to get a result on your computer, given that algorithms and codes have been written? Does it take several hours running? Thank you!

", "votes": null, "replies": [] }, { "id": 23638, "author_name": "eustache", "author_url": "", "post_date": "04/29/2013 16:38:37", "content": "

Mmm... what i was trying to say was that deep learning seems a paradigm of choice for this problem given its popularity in the recent years. Or so it seems from my limited knowledge of the semi supervised field. 

\r\n

 

\r\n

Of course if you or anyone else has pointers to other ss approaches that might work, i'm all ears ;-)

", "votes": null, "replies": [] }, { "id": 23639, "author_name": "iangoodfellow", "author_url": "", "post_date": "04/29/2013 16:41:09", "content": "

Since I'm organizing the contest I probably shouldn't provide advice for how to compete in it, beyond helping people troubleshoot pylearn2 stuff.

", "votes": null, "replies": [] }, { "id": 23645, "author_name": "eustache", "author_url": "", "post_date": "04/29/2013 17:00:06", "content": "

Of course that wasn't what I was asking !!!

\r\n

It's just that I was seeking new fields to explore and ideas ...

\r\n

if s.o. tells me I could look at the, say, manifold learning or co-training or self-training or ... he's not necessarily telling me how to solve the problem but rather educating me about existing, but possibly totally irrelevant approaches out there.\r\n

", "votes": null, "replies": [] }, { "id": 23683, "author_name": "rkirana", "author_url": "", "post_date": "04/30/2013 04:32:10", "content": "

is there any example on how to use Restricted boltzmann machines (rbm.py) with a sample .yaml file

", "votes": null, "replies": [] }, { "id": 23684, "author_name": "iangoodfellow", "author_url": "", "post_date": "04/30/2013 04:35:19", "content": "

pylearn2/scripts/tutorials/grbm_smd and pylearn2/scripts/tutorials/deep_trainer both use RBMs.

", "votes": null, "replies": [] }, { "id": 23687, "author_name": "rkirana", "author_url": "", "post_date": "04/30/2013 07:44:07", "content": "

that is for a binary RBM

\r\n

Can you let us know which function in rbm.py is a multi-class rbm?

", "votes": null, "replies": [] }, { "id": 23698, "author_name": "iangoodfellow", "author_url": "", "post_date": "04/30/2013 15:23:50", "content": "

Well, the hidden units are binary, but the visible units are real-valued.

\r\n

I don't think that rbm.py supports softmax variables, but dbm.py does. See pylearn2/scripts/tutorials/dbm_demo for a basic example of the interface.

", "votes": null, "replies": [] }, { "id": 23709, "author_name": "rkirana", "author_url": "", "post_date": "04/30/2013 16:19:39", "content": "

Thanks Ian.

\r\n

I guess the "y" layer must be softmax.

\r\n

rbm.yaml in the dbm_demo tutorial has no mention of the y interface

", "votes": null, "replies": [] }, { "id": 23711, "author_name": "iangoodfellow", "author_url": "", "post_date": "04/30/2013 16:23:56", "content": "

pylearn2.models.dbm.Softmax. There's not a nice flashy demo script for every feature of the library. If someone would like to make a pull request with more demos, that'd be great.

", "votes": null, "replies": [] }, { "id": 23712, "author_name": "rkirana", "author_url": "", "post_date": "04/30/2013 16:26:57", "content": "

ok, difficult for seasoned R programmers!

\r\n

visible_layer: !obj:pylearn2.models.dbm.BinaryVector

\r\n

to be replaced by

\r\n

visible_layer: !obj:pylearn2.models.dbm.Softmax

\r\n

 

\r\n

Correct?

", "votes": null, "replies": [] }, { "id": 23713, "author_name": "iangoodfellow", "author_url": "", "post_date": "04/30/2013 16:30:39", "content": "

Set the cost's supervised flag to True and use Softmax as the last hidden layer.

\r\n

The DBM code generally doesn't consider the class labels as "visible" because they're unobserved for test set data.

\r\n

The visible_layer for this challenge should be real-valued. I'm not sure if there's a real-valued DBM layer in the library right now.

", "votes": null, "replies": [] }, { "id": 23715, "author_name": "rkirana", "author_url": "", "post_date": "04/30/2013 17:03:33", "content": "

https://github.com/lisa-lab/pylearn2/tree/master/pylearn2/models/dbm

\r\n

There is no softmax here on github

", "votes": null, "replies": [] }, { "id": 23716, "author_name": "rkirana", "author_url": "", "post_date": "04/30/2013 17:16:10", "content": "

There is no softmax function in dbm.py also.

\r\n

It is only in mlp.py - Using same syntax gives following error:

\r\n

TypeError: Softmax does not support the following keywords: init_bias_target_marginals. Did you mean irange?

\r\n

!obj:pylearn2.models.dbm.Softmax {
\r\n                layer_name: 'h1',
\r\n                init_bias_target_marginals: *raw_train,
\r\n                # Initialize the weights to all 0s
\r\n                irange: .0,
\r\n                n_classes: 9
\r\n            }

\r\n


\r\n
\r\n

", "votes": null, "replies": [] }, { "id": 23717, "author_name": "iangoodfellow", "author_url": "", "post_date": "04/30/2013 17:17:48", "content": "

https://github.com/lisa-lab/pylearn2/blob/master/pylearn2/models/dbm/__init__.py line 1612.

", "votes": null, "replies": [] }, { "id": 23920, "author_name": "vnefedov", "author_url": "", "post_date": "05/05/2013 18:49:17", "content": "

My current algorithm (using python):

\r\n\r\n

Finally got 0.31 accuracy

", "votes": null, "replies": [] }, { "id": 23929, "author_name": "vnefedov", "author_url": "", "post_date": "05/06/2013 04:17:22", "content": "\r\n

0.35 accuracy

", "votes": null, "replies": [] }, { "id": 23937, "author_name": "ulrich2", "author_url": "", "post_date": "05/06/2013 12:16:35", "content": "

Just using Matlab NN toolbox without any pre processing. 

\r\n\r\n

0.40 accuracy

", "votes": null, "replies": [] }, { "id": 23988, "author_name": "potamitis", "author_url": "", "post_date": "05/07/2013 18:00:46", "content": "

[quote=Ulrich;23937]

\r\n

Just using Matlab NN toolbox without any pre processing. 

\r\n\r\n

0.40 accuracy

\r\n

[/quote]

\r\n

I run the Matlab NN toolbox and for more than 100 neurons I have 'out of memory' problem on 4GB machine. Did you face any problem as such?

", "votes": null, "replies": [] }, { "id": 23994, "author_name": "titericz", "author_url": "", "post_date": "05/07/2013 19:15:50", "content": "

Hi Rafael,

\r\n

 Try using net.trainFcn = 'traingdx' as the matlab train function instead of the default 'trainlm' and you can have more than 100 neurons... also try to use a small initial learning rate if you use 2 hidden layers...

", "votes": null, "replies": [] }, { "id": 24013, "author_name": "ccccat", "author_url": "", "post_date": "05/08/2013 14:01:55", "content": "

Also     net.efficiency.memoryReduction     parameter may help.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "22757": "", "22758": "", "22760": "", "22801": "", "22869": "", "22874": "", "22897": "", "22978": "", "22983": "", "22985": "", "23238": "", "23239": "", "23561": "", "23624": "", "23630": "", "23635": "", "23638": "", "23639": "", "23645": "", "23683": "", "23684": "", "23687": "", "23698": "", "23709": "", "23711": "", "23712": "", "23713": "", "23715": "", "23716": "", "23717": "", "23920": "", "23929": "", "23937": "", "23988": "", "23994": "", "24013": "" }, "source": "meta" }