{ "id": 15518, "title": "Wrote a NN based upon TINRTGU", "url": "/competitions/avito-context-ad-clicks/discussion/15518", "author_name": "", "post_date": "2015-07-24T18:02:38.503Z", "votes": 3, "comment_count": 6, "views": 2145, "content": "

Can someone sanity check this Python MLP with one hidden layer based upon TINRTGU? It seems to work but I think something is off and I haven't been able to successfully add hidden layers to it (my fault really). It doesn't seem to work really well compared to FTRLP though. Depending on features and data, scores around 0.045 compared to 0.043/0.044 for FTRLP.

", "messages": [ { "id": "86856", "postDate": "07/24/2015 18:02:38", "content": "

Can someone sanity check this Python MLP with one hidden layer based upon TINRTGU? It seems to work but I think something is off and I haven't been able to successfully add hidden layers to it (my fault really). It doesn't seem to work really well compared to FTRLP though. Depending on features and data, scores around 0.045 compared to 0.043/0.044 for FTRLP.

", "rawMarkdown": "Can someone sanity check this Python MLP with one hidden layer based upon TINRTGU? It seems to work but I think something is off and I haven't been able to successfully add hidden layers to it (my fault really). It doesn't seem to work really well compared to FTRLP though. Depending on features and data, scores around 0.045 compared to 0.043/0.044 for FTRLP.", "votes": null }, { "id": "86890", "postDate": "07/24/2015 23:48:00", "content": "

I might be wrong but have you tried symmetry breaking?

\n\n

The weights are initialized with zeros and there's no randomization -> Gradients at neurons from the same layer are identical and, consequently, neurons all over the layer have identical weights and produce the same output.

\n\n

Maybe its worth to try initializing randomly neurons in the hidden layer?

", "rawMarkdown": "I might be wrong but have you tried symmetry breaking?\r\n\r\nThe weights are initialized with zeros and there's no randomization -> Gradients at neurons from the same layer are identical and, consequently, neurons all over the layer have identical weights and produce the same output.\r\n\r\nMaybe its worth to try initializing randomly neurons in the hidden layer?", "votes": null }, { "id": "86910", "postDate": "07/25/2015 06:46:20", "content": "

I tried:\nimport random\nrandom.seed(1)\nsyn1 = [random.uniform(-1, 1) for j in xrange(H)] \nAnd it helps a little bit 0.0002 or so, but not much.

\n\n

I used this as a guide along with Tinrtgu: http://iamtrask.github.io/2015/07/12/basic-python-network/?

\n\n

I'm trying to figure out how to add another hidden layer, dropout, and regularize as well assuming the original code is okay.

", "rawMarkdown": "I tried:\r\nimport random\r\nrandom.seed(1)\r\nsyn1 = [random.uniform(-1, 1) for j in xrange(H)] \r\nAnd it helps a little bit 0.0002 or so, but not much. \r\n\r\nI used this as a guide along with Tinrtgu: http://iamtrask.github.io/2015/07/12/basic-python-network/?\r\n\r\nI'm trying to figure out how to add another hidden layer, dropout, and regularize as well assuming the original code is okay.", "votes": null }, { "id": "86955", "postDate": "07/25/2015 17:21:38", "content": "

I tried ftrl with nn in vowpal wabbit. Without success.

", "rawMarkdown": "I tried ftrl with nn in vowpal wabbit. Without success.", "votes": null }, { "id": "86977", "postDate": "07/25/2015 22:39:22", "content": "

I actually wrote this custom NN because I seem to be saturating or seg faulting Vowpal and I can't seem to get around it even with fairly moderate sub sampling or parameter adjustment. Oh well ... at least I gained some insight into neural nets.

", "rawMarkdown": "I actually wrote this custom NN because I seem to be saturating or seg faulting Vowpal and I can't seem to get around it even with fairly moderate sub sampling or parameter adjustment. Oh well ... at least I gained some insight into neural nets.", "votes": null }, { "id": "88103", "postDate": "08/03/2015 15:12:30", "content": "

Oh, i've just realized that there's one more python issue:

\n\n

syn0 = [w] * H #<- this line makes list of references, not copies

\n\n

That's why later all the neurons are updated instead of one.

\n\n

I'm terrible at explaining so here's an example:

\n\n

w = [.0]*2

\n\n

s = [w] * 3

\n\n

#s = [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0]]

\n\n

s[1][1] = .1

\n\n

#s == [[0.0, 0.1], [0.0, 0.1], [0.0, 0.1]]

", "rawMarkdown": "Oh, i've just realized that there's one more python issue:\r\n\r\nsyn0 = [w] * H #<- this line makes list of references, not copies\r\n\r\nThat's why later all the neurons are updated instead of one.\r\n\r\nI'm terrible at explaining so here's an example:\r\n\r\nw = [.0]*2\r\n\r\ns = [w] * 3\r\n\r\n\\#s = [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0]]\r\n\r\ns[1][1] = .1\r\n\r\n\\#s == [[0.0, 0.1], [0.0, 0.1], [0.0, 0.1]]", "votes": null }, { "id": "91544", "postDate": "09/04/2015 04:15:19", "content": "

[quote=thenx;88103]

\n\n

Oh, i've just realized that there's one more python issue:

\n\n

syn0 = [w] * H #<- this line makes list of references, not copies

\n\n

That's why later all the neurons are updated instead of one.

\n\n

I'm terrible at explaining so here's an example:

\n\n

w = [.0]*2

\n\n

s = [w] * 3

\n\n

#s = [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0]]

\n\n

s[1][1] = .1

\n\n

#s == [[0.0, 0.1], [0.0, 0.1], [0.0, 0.1]]

\n\n

[/quote]\nThis is correct. Thanks for catching this bug. I will fix the bug, but I don't have the merged data anymore.

", "rawMarkdown": "[quote=thenx;88103]\r\n\r\nOh, i've just realized that there's one more python issue:\r\n\r\nsyn0 = [w] * H #<- this line makes list of references, not copies\r\n\r\nThat's why later all the neurons are updated instead of one.\r\n\r\nI'm terrible at explaining so here's an example:\r\n\r\nw = [.0]*2\r\n\r\ns = [w] * 3\r\n\r\n\\#s = [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0]]\r\n\r\ns[1][1] = .1\r\n\r\n\\#s == [[0.0, 0.1], [0.0, 0.1], [0.0, 0.1]]\r\n\r\n[/quote]\r\nThis is correct. Thanks for catching this bug. I will fix the bug, but I don't have the merged data anymore.", "votes": null } ], "comments": [ { "id": 86890, "author_name": "thenx00", "author_url": "", "post_date": "07/24/2015 23:48:00", "content": "

I might be wrong but have you tried symmetry breaking?

\n\n

The weights are initialized with zeros and there's no randomization -> Gradients at neurons from the same layer are identical and, consequently, neurons all over the layer have identical weights and produce the same output.

\n\n

Maybe its worth to try initializing randomly neurons in the hidden layer?

", "votes": null, "replies": [] }, { "id": 86910, "author_name": "mikeskim", "author_url": "", "post_date": "07/25/2015 06:46:20", "content": "

I tried:\nimport random\nrandom.seed(1)\nsyn1 = [random.uniform(-1, 1) for j in xrange(H)] \nAnd it helps a little bit 0.0002 or so, but not much.

\n\n

I used this as a guide along with Tinrtgu: http://iamtrask.github.io/2015/07/12/basic-python-network/?

\n\n

I'm trying to figure out how to add another hidden layer, dropout, and regularize as well assuming the original code is okay.

", "votes": null, "replies": [] }, { "id": 86955, "author_name": "rushter", "author_url": "", "post_date": "07/25/2015 17:21:38", "content": "

I tried ftrl with nn in vowpal wabbit. Without success.

", "votes": null, "replies": [] }, { "id": 86977, "author_name": "mikeskim", "author_url": "", "post_date": "07/25/2015 22:39:22", "content": "

I actually wrote this custom NN because I seem to be saturating or seg faulting Vowpal and I can't seem to get around it even with fairly moderate sub sampling or parameter adjustment. Oh well ... at least I gained some insight into neural nets.

", "votes": null, "replies": [] }, { "id": 88103, "author_name": "thenx00", "author_url": "", "post_date": "08/03/2015 15:12:30", "content": "

Oh, i've just realized that there's one more python issue:

\n\n

syn0 = [w] * H #<- this line makes list of references, not copies

\n\n

That's why later all the neurons are updated instead of one.

\n\n

I'm terrible at explaining so here's an example:

\n\n

w = [.0]*2

\n\n

s = [w] * 3

\n\n

#s = [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0]]

\n\n

s[1][1] = .1

\n\n

#s == [[0.0, 0.1], [0.0, 0.1], [0.0, 0.1]]

", "votes": null, "replies": [] }, { "id": 91544, "author_name": "mikeskim", "author_url": "", "post_date": "09/04/2015 04:15:19", "content": "

[quote=thenx;88103]

\n\n

Oh, i've just realized that there's one more python issue:

\n\n

syn0 = [w] * H #<- this line makes list of references, not copies

\n\n

That's why later all the neurons are updated instead of one.

\n\n

I'm terrible at explaining so here's an example:

\n\n

w = [.0]*2

\n\n

s = [w] * 3

\n\n

#s = [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0]]

\n\n

s[1][1] = .1

\n\n

#s == [[0.0, 0.1], [0.0, 0.1], [0.0, 0.1]]

\n\n

[/quote]\nThis is correct. Thanks for catching this bug. I will fix the bug, but I don't have the merged data anymore.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "86856": "Can someone sanity check this Python MLP with one hidden layer based upon TINRTGU? It seems to work but I think something is off and I haven't been able to successfully add hidden layers to it (my fault really). It doesn't seem to work really well compared to FTRLP though. Depending on features and data, scores around 0.045 compared to 0.043/0.044 for FTRLP.", "86890": "I might be wrong but have you tried symmetry breaking?\r\n\r\nThe weights are initialized with zeros and there's no randomization -> Gradients at neurons from the same layer are identical and, consequently, neurons all over the layer have identical weights and produce the same output.\r\n\r\nMaybe its worth to try initializing randomly neurons in the hidden layer?", "86910": "I tried:\r\nimport random\r\nrandom.seed(1)\r\nsyn1 = [random.uniform(-1, 1) for j in xrange(H)] \r\nAnd it helps a little bit 0.0002 or so, but not much. \r\n\r\nI used this as a guide along with Tinrtgu: http://iamtrask.github.io/2015/07/12/basic-python-network/?\r\n\r\nI'm trying to figure out how to add another hidden layer, dropout, and regularize as well assuming the original code is okay.", "86955": "I tried ftrl with nn in vowpal wabbit. Without success.", "86977": "I actually wrote this custom NN because I seem to be saturating or seg faulting Vowpal and I can't seem to get around it even with fairly moderate sub sampling or parameter adjustment. Oh well ... at least I gained some insight into neural nets.", "88103": "Oh, i've just realized that there's one more python issue:\r\n\r\nsyn0 = [w] * H #<- this line makes list of references, not copies\r\n\r\nThat's why later all the neurons are updated instead of one.\r\n\r\nI'm terrible at explaining so here's an example:\r\n\r\nw = [.0]*2\r\n\r\ns = [w] * 3\r\n\r\n\\#s = [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0]]\r\n\r\ns[1][1] = .1\r\n\r\n\\#s == [[0.0, 0.1], [0.0, 0.1], [0.0, 0.1]]", "91544": "[quote=thenx;88103]\r\n\r\nOh, i've just realized that there's one more python issue:\r\n\r\nsyn0 = [w] * H #<- this line makes list of references, not copies\r\n\r\nThat's why later all the neurons are updated instead of one.\r\n\r\nI'm terrible at explaining so here's an example:\r\n\r\nw = [.0]*2\r\n\r\ns = [w] * 3\r\n\r\n\\#s = [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0]]\r\n\r\ns[1][1] = .1\r\n\r\n\\#s == [[0.0, 0.1], [0.0, 0.1], [0.0, 0.1]]\r\n\r\n[/quote]\r\nThis is correct. Thanks for catching this bug. I will fix the bug, but I don't have the merged data anymore." }, "source": "meta" }