{
  "id": 4403,
  "title": "MLP with sigmoid units and three hidden layers",
  "url": "/competitions/challenges-in-representation-learning-the-black-box-learning-challenge/discussion/4403",
  "author_name": "",
  "post_date": "2013-04-21T20:20:24.553Z",
  "votes": null,
  "comment_count": 23,
  "views": 9944,
  "content": "<p>For the &quot;MLP with sigmoid units and three hidden layers&quot; becnhmark, how many units do each of the three hidden layers have? &nbsp;I'm trying to replicate this benchmark, without much success.</p>",
  "messages": [
    {
      "id": "23299",
      "postDate": "04/21/2013 20:20:24",
      "content": "<p>For the &quot;MLP with sigmoid units and three hidden layers&quot; becnhmark, how many units do each of the three hidden layers have? &nbsp;I'm trying to replicate this benchmark, without much success.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23307",
      "postDate": "04/21/2013 22:55:45",
      "content": "<p>1000 each! Good luck :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23692",
      "postDate": "04/30/2013 12:06:38",
      "content": "<p>[quote=Dumitru;23307]</p>\r\n<p>1000 each! Good luck :)</p>\r\n<p>[/quote]</p>\r\n<p>Is 1000 a default value?</p>\r\n<p>In R nnet units choices are in 10-100 range. For 11.000 samples (train &#43; test) I need 26Gb of RAM for training only 35 units. &nbsp;I see in pylearn2 the units are one/two orders of magnitude higher.</p>\r\n<p>Don't know if this is for a different technical approach, with a scientific basis, or only because pylearn2 is faster and don't needs so much memory than R nnet (thinking that regularization ensures shrinking the weights of unnecessary units towards 0).\r\n &nbsp;</p>\r\n<p>I'm finding nnet hard to use with a moderate sized dataset. I'm thinking in give a chance to pylearn2.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23695",
      "postDate": "04/30/2013 13:58:31",
      "content": "<p>Jose,</p>\r\n<p>R's nnet package was originally written a long time ago, and there's been a shift towards the use of bigger neural networks (both in terms of more hidden layers, and more units in each hidden layer).&nbsp; This trend has been especially dramatic recently with\r\n the success of &quot;deep learning.&quot;&nbsp; This shift has been driven by improvements in both hardware and in the underlying algorithms.&nbsp;</p>\r\n<p>As an example of &quot;algorithmic change&quot;, larger networks are more sensitive to how the weights are initialized, and our knowledge about how to initialize weights has improved over time.</p>\r\n<p>Similarly, larger datasets allow more precise estimation of the weights in large networks.&nbsp; So the ideal network topology for a large dataset will have more units than the ideal topology for a smaller dataset.</p>\r\n<p>This competition gives a reasonably large amount of unlabeled data, and modern NN techniques allow unlabeled data to be incorporated in training (e.g. using autoencoders).&nbsp; As a result, larger networks seem to work well for this problem.</p>\r\n<p>I think pylearn2 caters more to this type of problem, whereas the nnet package in R is intended for problems that don't, for example, leverage unlabeled data.&nbsp; As a result, the sizes of networks used are going to be much different.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23697",
      "postDate": "04/30/2013 15:11:36",
      "content": "<p>I've never used any of the half-dozen R neural network packages that show up via a quick CPAN search, so I can't really speak intelligently about them.&nbsp; From the descriptions I suspect most are fairly old.</p>\r\n<p>For what it's worth, there are some fairly modern open source neural network libraries out there.&nbsp; You might want to investigate FANN or OpenNN for example, neither of which I've used unmodified, but both of which I've &quot;borrowed&quot; from for my own purposes\r\n over the past couple of years.&nbsp; I don't remember for certain, but I think FANN might have an R binding (not published to CPAN, though).</p>\r\n<p>It does seem clear at this point that neural networks seem to perform better than simpler methods for this challenge, and as soon as I have time (human and compute) I'll throw my own novel NN code at the problem and see what happens.&nbsp;\r\n</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23699",
      "postDate": "04/30/2013 15:33:13",
      "content": "<p>Suppose you use a neural net with 1,000 input units and 1,000 output units. The weight matrix thus has 1,000 * 1,000 = 1e6 elements. Training the network will require also computing the gradient on the weights, so you'll need 2e6 numbers there. There's also\r\n the state of the network, and the gradient of the state of the network-- around 4,000 numbers per example, so 400,000 numbers if you use a minibatch of size 100. All in all, under 3e6 numbers per layer. So let's say you use 4 layers. That will need less than\r\n 12e6 (actually a lot less, because just multiplying like that double-counts most of the state). Even if you use 64 bit precision, 12e6 * 8 / (1024^3) = .08. You shouldn't need even 1/10th of 1 gigabyte to train a multilayer network with 1,000 units per layer.</p>\r\n<p>José, I don't have any experience with R or its neural net libraries. It's possible that you're configuring R or the neural net library incorrectly, or it's possible that R / the neural net library is just incredibly wasteful of memory. In any case, if you\r\n use a good neural net implementation correctly, you should be able to train much larger networks than we included in the demos.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23701",
      "postDate": "04/30/2013 15:39:41",
      "content": "<p>I tried the <a href=\"http://cran.r-project.org/web/packages/RSNNS/index.html\">\r\nRSNNS</a> package, which is an interface to the&nbsp;<a href=\"http://www.ra.cs.uni-tuebingen.de/SNNS/\">Stuttgart Neural Network Simulator</a>, which is more modern than nnet, but still pretty old.</p>\r\n<p>I was able to train an mlp with 3 layers and 500 units per layer, which got an accuracy of about .36 in my internal cross-validation. &nbsp;I bumped it up to 750 units per layer, and accuracy dropped down to about .21. &nbsp;Then I tried 1000 units per layer, and\r\n my computer crashed.</p>\r\n<p>It seems like R doesn't have any packages for the more modern neural network libraries. If you really want to use R, I think you'll have better luck with RSNSS for this competition.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23718",
      "postDate": "04/30/2013 17:47:29",
      "content": "<p>Thank you all.</p>\r\n<p>Seems that for NN, R isn't the way.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23720",
      "postDate": "04/30/2013 18:15:28",
      "content": "<p>The problem here is the training procedure. the R and Matlabs nnet uses gradient descent to update the weights, the pylearn2 uses stochastic gradient descent. With low learning rates and a good batch size, you can prevent overfiting using early stop.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23724",
      "postDate": "04/30/2013 19:27:38",
      "content": "<p>Zach:<br>\r\nyou are right.</p>\r\n<p>RSNNS gives worse results than nnet. Infact neuralnet is also worse than nnet.</p>\r\n<p>nnet is the best neural network package in R and is outdated. Only package close to nnet in R is AMORE</p>\r\n<p><br>\r\nThanks<br>\r\nkiran</p>\r\n<p>&nbsp;</p>\r\n<p>[quote=Zach;23701]</p>\r\n<p>I tried the <a href=\"http://cran.r-project.org/web/packages/RSNNS/index.html\">\r\nRSNNS</a> package, which is an interface to the&nbsp;<a href=\"http://www.ra.cs.uni-tuebingen.de/SNNS/\">Stuttgart Neural Network Simulator</a>, which is more modern than nnet, but still pretty old.</p>\r\n<p>I was able to train an mlp with 3 layers and 500 units per layer, which got an accuracy of about .36 in my internal cross-validation. &nbsp;I bumped it up to 750 units per layer, and accuracy dropped down to about .21. &nbsp;Then I tried 1000 units per layer, and\r\n my computer crashed.</p>\r\n<p>It seems like R doesn't have any packages for the more modern neural network libraries. If you really want to use R, I think you'll have better luck with RSNSS for this competition.</p>\r\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23725",
      "postDate": "04/30/2013 19:28:23",
      "content": "<p>[quote=YetiMan;23697]</p>\r\n<p>I've never used any of the half-dozen R neural network packages that show up via a quick CPAN search, so I can't really speak intelligently about them.&nbsp; From the descriptions I suspect most are fairly old.</p>\r\n<p>For what it's worth, there are some fairly modern open source neural network libraries out there.&nbsp; You might want to investigate FANN or OpenNN for example, neither of which I've used unmodified, but both of which I've &quot;borrowed&quot; from for my own purposes\r\n over the past couple of years.&nbsp; I don't remember for certain, but I think FANN might have an R binding (not published to CPAN, though).</p>\r\n<p>It does seem clear at this point that neural networks seem to perform better than simpler methods for this challenge, and as soon as I have time (human and compute) I'll throw my own novel NN code at the problem and see what happens.&nbsp;</p>\r\n<p>[/quote]</p>\r\n<p>yes FANN is in R</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23726",
      "postDate": "04/30/2013 20:03:33",
      "content": "<p>[quote=Black Magic;23725]</p>\r\n<p>yes FANN is in R</p>\r\n<p>[/quote]</p>\r\n<p>Is there a package or a guide out there for FANN in R?</p>\r\n<p>[quote=Black Magic;23724]</p>\r\n<p>Zach:<br>\r\nyou are right.</p>\r\n<p>RSNNS gives worse results than nnet. Infact neuralnet is also worse than nnet.</p>\r\n<p>nnet is the best neural network package in R and is outdated. Only package close to nnet in R is AMORE</p>\r\n<p>[/quote]</p>\r\n<p>Do you have some evidence or references for this? &nbsp;I use nnet a lot, but in many cases (including this competition), I've found RSNNS to yield better models.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23727",
      "postDate": "04/30/2013 20:07:25",
      "content": "<p>[quote=Zach;23726]</p>\r\n<p>Is there a package or a guide out there for FANN in R?</p>\r\n<p>[quote=Black Magic;23724]</p>\r\n<p>It appears to be here: http://sourceforge.net/projects/rfann/</p>\r\n<p>But I know nothing about it.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23728",
      "postDate": "04/30/2013 20:10:57",
      "content": "<p>Yes, using nnet you can get 0.45 CV here without preprocessing - &nbsp;[16 hidden nodes, 1 layer[</p>\r\n<p>with RSNNS I could not beat it. amore could get close to what nnet got.</p>\r\n<p>What is the cv score you got using RSNNS - depends on the params also maybe</p>\r\n<p>&nbsp;</p>\r\n<p>[quote=Zach;23726]</p>\r\n<p>[quote=Black Magic;23725]</p>\r\n<p>yes FANN is in R</p>\r\n<p>[/quote]</p>\r\n<p>Is there a package or a guide out there for FANN in R?</p>\r\n<p>[quote=Black Magic;23724]</p>\r\n<p>Zach:<br>\r\nyou are right.</p>\r\n<p>RSNNS gives worse results than nnet. Infact neuralnet is also worse than nnet.</p>\r\n<p>nnet is the best neural network package in R and is outdated. Only package close to nnet in R is AMORE</p>\r\n<p>[/quote]</p>\r\n<p>Do you have some evidence or references for this? &nbsp;I use nnet a lot, but in many cases (including this competition), I've found RSNNS to yield better models.</p>\r\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23729",
      "postDate": "04/30/2013 20:19:03",
      "content": "<p>[quote=Black Magic;23728]</p>\r\n<p>Yes, using nnet you can get 0.45 CV here without preprocessing - &nbsp;[16 hidden nodes, 1 layer[</p>\r\n<p>with RSNNS I could not beat it. amore could get close to what nnet got.</p>\r\n<p>What is the cv score you got using RSNNS - depends on the params also maybe</p>\r\n<p>[/quote]</p>\r\n<p>&nbsp;</p>\r\n<p>Wow, I have to try that. &nbsp;With the mlp function in RSNNS, my CV maxed out at 0.36, with 3 layers, 500 nodes per layer.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23732",
      "postDate": "04/30/2013 22:15:56",
      "content": "<div>I too am hard up against the limits of Rs NN capabilities. I'm a bit out of practice, but I can comment that my experiments with RSNNS consistently perform worse than experiments with nnet - which I find frustrating because RSSNSs mlp seems more flexible\r\n in terms of methods and topology.&nbsp;</div>\r\n<div>Working with datasets with 50-100 features (after feature selection) I can achieve scores of 0.45-0.48 with nnet configured with 30-70 nodes and weight decay of 0.1 (10 fold cross validation, 5 repeats using caret). This method in this harness produces\r\n scores that seem to correlate highly with the leaderboard compared to many regularization-based methods I've attempted.</div>\r\n<div>Hat in hand, I think I'll be wandering over to pylearn2 this weekend.</div>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23733",
      "postDate": "04/30/2013 22:59:51",
      "content": "<p>I'm only using R thus far and have had relatively good results just self-training some simple models (my leaderboard has me at ~0.58 as I write this). I don't think that this approach can lead to win in this competetion, but it gives me hope that I won't\r\n have to use some non-R language to do moderately well.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23734",
      "postDate": "04/30/2013 23:07:12",
      "content": "<p>[quote=Leustagos;23720]</p>\r\n<p>The problem here is the training procedure. the R and Matlabs nnet uses gradient descent to update the weights, the pylearn2 uses stochastic gradient descent. With low learning rates and a good batch size, you can prevent overfiting using early stop.</p>\r\n<p>[/quote]</p>\r\n<p>Does anybody have any (any !) success with Matlab Neural Networks? I did not get anything better than 0.3 on CV even with several layers of 1000 neurons.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23735",
      "postDate": "04/30/2013 23:09:02",
      "content": "<p>i got 0.56 on the leaderboard with matlab nnet.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23736",
      "postDate": "04/30/2013 23:39:29",
      "content": "<p>[quote=Leustagos;23735]</p>\r\n<p>i got 0.56 on the leaderboard with matlab nnet.</p>\r\n<p>[/quote]</p>\r\n<p>&nbsp;</p>\r\n<p>Thank you. It sounds very encouraging.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23750",
      "postDate": "05/01/2013 15:01:43",
      "content": "<p>nnet - goes upto 0.46 without data preparation</p>\r\n<p>amore - matches nnet - but max of 0.44</p>\r\n<p>RSNNS - I could get upto 0.38</p>\r\n<p>neuralnet - never more than 0.34</p>\r\n<p>&nbsp;</p>\r\n<p>That is the story of the R packages</p>\r\n<p>&nbsp;</p>\r\n<p>[quote=Zach;23729]</p>\r\n<p>[quote=Black Magic;23728]</p>\r\n<p>Yes, using nnet you can get 0.45 CV here without preprocessing - &nbsp;[16 hidden nodes, 1 layer[</p>\r\n<p>with RSNNS I could not beat it. amore could get close to what nnet got.</p>\r\n<p>What is the cv score you got using RSNNS - depends on the params also maybe</p>\r\n<p>[/quote]</p>\r\n<p>&nbsp;</p>\r\n<p>Wow, I have to try that. &nbsp;With the mlp function in RSNNS, my CV maxed out at 0.36, with 3 layers, 500 nodes per layer.</p>\r\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23757",
      "postDate": "05/01/2013 17:30:32",
      "content": "<p>I tried using nnet (though I'm not used to working with MLPs) and it kept saying too many weights so couldn't run.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23758",
      "postDate": "05/01/2013 17:33:49",
      "content": "<p>Domcastro: &nbsp;You can make it run by setting a sufficiently high MaxNWts when you call nnet.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23759",
      "postDate": "05/01/2013 17:53:26",
      "content": "<p>ah yes - silly me. :)</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 23307,
      "author_name": "dumitru0",
      "author_url": "",
      "post_date": "04/21/2013 22:55:45",
      "content": "<p>1000 each! Good luck :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23692,
      "author_name": "blindape",
      "author_url": "",
      "post_date": "04/30/2013 12:06:38",
      "content": "<p>[quote=Dumitru;23307]</p>\r\n<p>1000 each! Good luck :)</p>\r\n<p>[/quote]</p>\r\n<p>Is 1000 a default value?</p>\r\n<p>In R nnet units choices are in 10-100 range. For 11.000 samples (train &#43; test) I need 26Gb of RAM for training only 35 units. &nbsp;I see in pylearn2 the units are one/two orders of magnitude higher.</p>\r\n<p>Don't know if this is for a different technical approach, with a scientific basis, or only because pylearn2 is faster and don't needs so much memory than R nnet (thinking that regularization ensures shrinking the weights of unnecessary units towards 0).\r\n &nbsp;</p>\r\n<p>I'm finding nnet hard to use with a moderate sized dataset. I'm thinking in give a chance to pylearn2.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23695,
      "author_name": "dansbecker",
      "author_url": "",
      "post_date": "04/30/2013 13:58:31",
      "content": "<p>Jose,</p>\r\n<p>R's nnet package was originally written a long time ago, and there's been a shift towards the use of bigger neural networks (both in terms of more hidden layers, and more units in each hidden layer).&nbsp; This trend has been especially dramatic recently with\r\n the success of &quot;deep learning.&quot;&nbsp; This shift has been driven by improvements in both hardware and in the underlying algorithms.&nbsp;</p>\r\n<p>As an example of &quot;algorithmic change&quot;, larger networks are more sensitive to how the weights are initialized, and our knowledge about how to initialize weights has improved over time.</p>\r\n<p>Similarly, larger datasets allow more precise estimation of the weights in large networks.&nbsp; So the ideal network topology for a large dataset will have more units than the ideal topology for a smaller dataset.</p>\r\n<p>This competition gives a reasonably large amount of unlabeled data, and modern NN techniques allow unlabeled data to be incorporated in training (e.g. using autoencoders).&nbsp; As a result, larger networks seem to work well for this problem.</p>\r\n<p>I think pylearn2 caters more to this type of problem, whereas the nnet package in R is intended for problems that don't, for example, leverage unlabeled data.&nbsp; As a result, the sizes of networks used are going to be much different.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23697,
      "author_name": "yetiman",
      "author_url": "",
      "post_date": "04/30/2013 15:11:36",
      "content": "<p>I've never used any of the half-dozen R neural network packages that show up via a quick CPAN search, so I can't really speak intelligently about them.&nbsp; From the descriptions I suspect most are fairly old.</p>\r\n<p>For what it's worth, there are some fairly modern open source neural network libraries out there.&nbsp; You might want to investigate FANN or OpenNN for example, neither of which I've used unmodified, but both of which I've &quot;borrowed&quot; from for my own purposes\r\n over the past couple of years.&nbsp; I don't remember for certain, but I think FANN might have an R binding (not published to CPAN, though).</p>\r\n<p>It does seem clear at this point that neural networks seem to perform better than simpler methods for this challenge, and as soon as I have time (human and compute) I'll throw my own novel NN code at the problem and see what happens.&nbsp;\r\n</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23699,
      "author_name": "iangoodfellow",
      "author_url": "",
      "post_date": "04/30/2013 15:33:13",
      "content": "<p>Suppose you use a neural net with 1,000 input units and 1,000 output units. The weight matrix thus has 1,000 * 1,000 = 1e6 elements. Training the network will require also computing the gradient on the weights, so you'll need 2e6 numbers there. There's also\r\n the state of the network, and the gradient of the state of the network-- around 4,000 numbers per example, so 400,000 numbers if you use a minibatch of size 100. All in all, under 3e6 numbers per layer. So let's say you use 4 layers. That will need less than\r\n 12e6 (actually a lot less, because just multiplying like that double-counts most of the state). Even if you use 64 bit precision, 12e6 * 8 / (1024^3) = .08. You shouldn't need even 1/10th of 1 gigabyte to train a multilayer network with 1,000 units per layer.</p>\r\n<p>José, I don't have any experience with R or its neural net libraries. It's possible that you're configuring R or the neural net library incorrectly, or it's possible that R / the neural net library is just incredibly wasteful of memory. In any case, if you\r\n use a good neural net implementation correctly, you should be able to train much larger networks than we included in the demos.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23701,
      "author_name": "zachmayer",
      "author_url": "",
      "post_date": "04/30/2013 15:39:41",
      "content": "<p>I tried the <a href=\"http://cran.r-project.org/web/packages/RSNNS/index.html\">\r\nRSNNS</a> package, which is an interface to the&nbsp;<a href=\"http://www.ra.cs.uni-tuebingen.de/SNNS/\">Stuttgart Neural Network Simulator</a>, which is more modern than nnet, but still pretty old.</p>\r\n<p>I was able to train an mlp with 3 layers and 500 units per layer, which got an accuracy of about .36 in my internal cross-validation. &nbsp;I bumped it up to 750 units per layer, and accuracy dropped down to about .21. &nbsp;Then I tried 1000 units per layer, and\r\n my computer crashed.</p>\r\n<p>It seems like R doesn't have any packages for the more modern neural network libraries. If you really want to use R, I think you'll have better luck with RSNSS for this competition.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23718,
      "author_name": "blindape",
      "author_url": "",
      "post_date": "04/30/2013 17:47:29",
      "content": "<p>Thank you all.</p>\r\n<p>Seems that for NN, R isn't the way.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23720,
      "author_name": "leustagos",
      "author_url": "",
      "post_date": "04/30/2013 18:15:28",
      "content": "<p>The problem here is the training procedure. the R and Matlabs nnet uses gradient descent to update the weights, the pylearn2 uses stochastic gradient descent. With low learning rates and a good batch size, you can prevent overfiting using early stop.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23724,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "04/30/2013 19:27:38",
      "content": "<p>Zach:<br>\r\nyou are right.</p>\r\n<p>RSNNS gives worse results than nnet. Infact neuralnet is also worse than nnet.</p>\r\n<p>nnet is the best neural network package in R and is outdated. Only package close to nnet in R is AMORE</p>\r\n<p><br>\r\nThanks<br>\r\nkiran</p>\r\n<p>&nbsp;</p>\r\n<p>[quote=Zach;23701]</p>\r\n<p>I tried the <a href=\"http://cran.r-project.org/web/packages/RSNNS/index.html\">\r\nRSNNS</a> package, which is an interface to the&nbsp;<a href=\"http://www.ra.cs.uni-tuebingen.de/SNNS/\">Stuttgart Neural Network Simulator</a>, which is more modern than nnet, but still pretty old.</p>\r\n<p>I was able to train an mlp with 3 layers and 500 units per layer, which got an accuracy of about .36 in my internal cross-validation. &nbsp;I bumped it up to 750 units per layer, and accuracy dropped down to about .21. &nbsp;Then I tried 1000 units per layer, and\r\n my computer crashed.</p>\r\n<p>It seems like R doesn't have any packages for the more modern neural network libraries. If you really want to use R, I think you'll have better luck with RSNSS for this competition.</p>\r\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23725,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "04/30/2013 19:28:23",
      "content": "<p>[quote=YetiMan;23697]</p>\r\n<p>I've never used any of the half-dozen R neural network packages that show up via a quick CPAN search, so I can't really speak intelligently about them.&nbsp; From the descriptions I suspect most are fairly old.</p>\r\n<p>For what it's worth, there are some fairly modern open source neural network libraries out there.&nbsp; You might want to investigate FANN or OpenNN for example, neither of which I've used unmodified, but both of which I've &quot;borrowed&quot; from for my own purposes\r\n over the past couple of years.&nbsp; I don't remember for certain, but I think FANN might have an R binding (not published to CPAN, though).</p>\r\n<p>It does seem clear at this point that neural networks seem to perform better than simpler methods for this challenge, and as soon as I have time (human and compute) I'll throw my own novel NN code at the problem and see what happens.&nbsp;</p>\r\n<p>[/quote]</p>\r\n<p>yes FANN is in R</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23726,
      "author_name": "zachmayer",
      "author_url": "",
      "post_date": "04/30/2013 20:03:33",
      "content": "<p>[quote=Black Magic;23725]</p>\r\n<p>yes FANN is in R</p>\r\n<p>[/quote]</p>\r\n<p>Is there a package or a guide out there for FANN in R?</p>\r\n<p>[quote=Black Magic;23724]</p>\r\n<p>Zach:<br>\r\nyou are right.</p>\r\n<p>RSNNS gives worse results than nnet. Infact neuralnet is also worse than nnet.</p>\r\n<p>nnet is the best neural network package in R and is outdated. Only package close to nnet in R is AMORE</p>\r\n<p>[/quote]</p>\r\n<p>Do you have some evidence or references for this? &nbsp;I use nnet a lot, but in many cases (including this competition), I've found RSNNS to yield better models.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23727,
      "author_name": "yetiman",
      "author_url": "",
      "post_date": "04/30/2013 20:07:25",
      "content": "<p>[quote=Zach;23726]</p>\r\n<p>Is there a package or a guide out there for FANN in R?</p>\r\n<p>[quote=Black Magic;23724]</p>\r\n<p>It appears to be here: http://sourceforge.net/projects/rfann/</p>\r\n<p>But I know nothing about it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23728,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "04/30/2013 20:10:57",
      "content": "<p>Yes, using nnet you can get 0.45 CV here without preprocessing - &nbsp;[16 hidden nodes, 1 layer[</p>\r\n<p>with RSNNS I could not beat it. amore could get close to what nnet got.</p>\r\n<p>What is the cv score you got using RSNNS - depends on the params also maybe</p>\r\n<p>&nbsp;</p>\r\n<p>[quote=Zach;23726]</p>\r\n<p>[quote=Black Magic;23725]</p>\r\n<p>yes FANN is in R</p>\r\n<p>[/quote]</p>\r\n<p>Is there a package or a guide out there for FANN in R?</p>\r\n<p>[quote=Black Magic;23724]</p>\r\n<p>Zach:<br>\r\nyou are right.</p>\r\n<p>RSNNS gives worse results than nnet. Infact neuralnet is also worse than nnet.</p>\r\n<p>nnet is the best neural network package in R and is outdated. Only package close to nnet in R is AMORE</p>\r\n<p>[/quote]</p>\r\n<p>Do you have some evidence or references for this? &nbsp;I use nnet a lot, but in many cases (including this competition), I've found RSNNS to yield better models.</p>\r\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23729,
      "author_name": "zachmayer",
      "author_url": "",
      "post_date": "04/30/2013 20:19:03",
      "content": "<p>[quote=Black Magic;23728]</p>\r\n<p>Yes, using nnet you can get 0.45 CV here without preprocessing - &nbsp;[16 hidden nodes, 1 layer[</p>\r\n<p>with RSNNS I could not beat it. amore could get close to what nnet got.</p>\r\n<p>What is the cv score you got using RSNNS - depends on the params also maybe</p>\r\n<p>[/quote]</p>\r\n<p>&nbsp;</p>\r\n<p>Wow, I have to try that. &nbsp;With the mlp function in RSNNS, my CV maxed out at 0.36, with 3 layers, 500 nodes per layer.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23732,
      "author_name": "jasonbrownlee",
      "author_url": "",
      "post_date": "04/30/2013 22:15:56",
      "content": "<div>I too am hard up against the limits of Rs NN capabilities. I'm a bit out of practice, but I can comment that my experiments with RSNNS consistently perform worse than experiments with nnet - which I find frustrating because RSSNSs mlp seems more flexible\r\n in terms of methods and topology.&nbsp;</div>\r\n<div>Working with datasets with 50-100 features (after feature selection) I can achieve scores of 0.45-0.48 with nnet configured with 30-70 nodes and weight decay of 0.1 (10 fold cross validation, 5 repeats using caret). This method in this harness produces\r\n scores that seem to correlate highly with the leaderboard compared to many regularization-based methods I've attempted.</div>\r\n<div>Hat in hand, I think I'll be wandering over to pylearn2 this weekend.</div>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23733,
      "author_name": "dmcgarry",
      "author_url": "",
      "post_date": "04/30/2013 22:59:51",
      "content": "<p>I'm only using R thus far and have had relatively good results just self-training some simple models (my leaderboard has me at ~0.58 as I write this). I don't think that this approach can lead to win in this competetion, but it gives me hope that I won't\r\n have to use some non-R language to do moderately well.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23734,
      "author_name": "ccccat",
      "author_url": "",
      "post_date": "04/30/2013 23:07:12",
      "content": "<p>[quote=Leustagos;23720]</p>\r\n<p>The problem here is the training procedure. the R and Matlabs nnet uses gradient descent to update the weights, the pylearn2 uses stochastic gradient descent. With low learning rates and a good batch size, you can prevent overfiting using early stop.</p>\r\n<p>[/quote]</p>\r\n<p>Does anybody have any (any !) success with Matlab Neural Networks? I did not get anything better than 0.3 on CV even with several layers of 1000 neurons.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23735,
      "author_name": "leustagos",
      "author_url": "",
      "post_date": "04/30/2013 23:09:02",
      "content": "<p>i got 0.56 on the leaderboard with matlab nnet.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23736,
      "author_name": "ccccat",
      "author_url": "",
      "post_date": "04/30/2013 23:39:29",
      "content": "<p>[quote=Leustagos;23735]</p>\r\n<p>i got 0.56 on the leaderboard with matlab nnet.</p>\r\n<p>[/quote]</p>\r\n<p>&nbsp;</p>\r\n<p>Thank you. It sounds very encouraging.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23750,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "05/01/2013 15:01:43",
      "content": "<p>nnet - goes upto 0.46 without data preparation</p>\r\n<p>amore - matches nnet - but max of 0.44</p>\r\n<p>RSNNS - I could get upto 0.38</p>\r\n<p>neuralnet - never more than 0.34</p>\r\n<p>&nbsp;</p>\r\n<p>That is the story of the R packages</p>\r\n<p>&nbsp;</p>\r\n<p>[quote=Zach;23729]</p>\r\n<p>[quote=Black Magic;23728]</p>\r\n<p>Yes, using nnet you can get 0.45 CV here without preprocessing - &nbsp;[16 hidden nodes, 1 layer[</p>\r\n<p>with RSNNS I could not beat it. amore could get close to what nnet got.</p>\r\n<p>What is the cv score you got using RSNNS - depends on the params also maybe</p>\r\n<p>[/quote]</p>\r\n<p>&nbsp;</p>\r\n<p>Wow, I have to try that. &nbsp;With the mlp function in RSNNS, my CV maxed out at 0.36, with 3 layers, 500 nodes per layer.</p>\r\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23757,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "05/01/2013 17:30:32",
      "content": "<p>I tried using nnet (though I'm not used to working with MLPs) and it kept saying too many weights so couldn't run.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23758,
      "author_name": "dansbecker",
      "author_url": "",
      "post_date": "05/01/2013 17:33:49",
      "content": "<p>Domcastro: &nbsp;You can make it run by setting a sufficiently high MaxNWts when you call nnet.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23759,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "05/01/2013 17:53:26",
      "content": "<p>ah yes - silly me. :)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "23299": "",
    "23307": "",
    "23692": "",
    "23695": "",
    "23697": "",
    "23699": "",
    "23701": "",
    "23718": "",
    "23720": "",
    "23724": "",
    "23725": "",
    "23726": "",
    "23727": "",
    "23728": "",
    "23729": "",
    "23732": "",
    "23733": "",
    "23734": "",
    "23735": "",
    "23736": "",
    "23750": "",
    "23757": "",
    "23758": "",
    "23759": ""
  },
  "source": "meta"
}