[quote=Sudeep Juvekar;100988]
\n\nAny plans of supporting older GPUs/GPUs with compute capabilities < 5.0 on Neon?
\n\nMost of us can only get hold of grid K520 on EC2, for example.
\n\n[/quote]
\n\nI have the same question
", "rawMarkdown": "[quote=Sudeep Juvekar;100988]\r\n\r\nAny plans of supporting older GPUs/GPUs with compute capabilities < 5.0 on Neon? \r\n\r\nMost of us can only get hold of grid K520 on EC2, for example. \r\n\r\n[/quote]\r\n\r\nI have the same question", "votes": 1 }, { "id": 101115, "postDate": "2015-12-14T02:51:19.050Z", "content": "Hold on! Found a bug in the code. It turns out that the probability of getting a 3.5ish score in a single try is less than 0.25. Will push an update tomorrow that should make it more reliable.
", "rawMarkdown": "Hold on! Found a bug in the code. It turns out that the probability of getting a 3.5ish score in a single try is less than 0.25. Will push an update tomorrow that should make it more reliable.", "votes": 1 }, { "id": 100988, "postDate": "2015-12-12T17:24:13.313Z", "content": "Any plans of supporting older GPUs/GPUs with compute capabilities < 5.0 on Neon?
\n\nMost of us can only get hold of grid K520 on EC2, for example.
", "rawMarkdown": "Any plans of supporting older GPUs/GPUs with compute capabilities < 5.0 on Neon? \r\n\r\nMost of us can only get hold of grid K520 on EC2, for example. ", "votes": 1 }, { "id": 100262, "postDate": "2015-12-04T09:15:25.760Z", "content": "Hello Anil,\nOne month ago I took roughly the same approach. I still have pain on my wrist from all the clicking :)
\nAlthough I did not do an encoder ( I regressed) my mugshots look roughly the same.
\nHowever, I abondoned the "project" since my holdout loss was roughly 5.4 so I thought that I was waayyy off with my approach.
My classifier of the mughots to get the final predictions was just not good.
\nNow looking at your presentation it seems like ths final classifier is just a formality..
\nFew layers, bit of dropout et voila.. However.. For me it seems like that part is the big kicker..
Could you assess if my mugshots are worse then yours ? That might be the reason the my classifier did not work. Or might my neural network classifier not be up to par with your Neon classifier. (Which looks very interesting).
Edit.. There was a bug in my labeling code .. Excuse me
\n\nMugs of my holdout are attached.
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/100262/3356/mugs.png?sv=2012-02-12&se=2015-12-08T10:04:50Z&sr=b&sp=r&sig=fqNTJBwP4VLU42xzwpBZGI4XrlrGy0B3bdOAsf%2BreJI%3D\")
\r\nAlthough I did not do an encoder ( I regressed) my mugshots look roughly the same.
\r\nHowever, I abondoned the \"project\" since my holdout loss was roughly 5.4 so I thought that I was waayyy off with my approach.
\r\n\r\nMy classifier of the mughots to get the final predictions was just not good.
\r\nNow looking at your presentation it seems like ths final classifier is just a formality..
\r\nFew layers, bit of dropout et voila.. However.. For me it seems like that part is the big kicker..
\r\n\r\nCould you assess if my mugshots are worse then yours ? That might be the reason the my classifier did not work. Or might my neural network classifier not be up to par with your Neon classifier. (Which looks very interesting).
\r\n\r\n**Edit.. There was a bug in my labeling code .. Excuse me**\r\n\r\nMugs of my holdout are attached.\r\n\r\n![enter image description here][1]\r\n\r\n\r\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/100262/3356/mugs.png?sv=2012-02-12&se=2015-12-08T10%3A04%3A50Z&sr=b&sp=r&sig=fqNTJBwP4VLU42xzwpBZGI4XrlrGy0B3bdOAsf%2BreJI%3D", "votes": 1 }, { "id": 99444, "postDate": "2015-11-24T21:57:16.473Z", "content": "
[quote=Sudeep Juvekar;99439]
\n\nDo we have permission to use the bonnet-tip and blowhead (point1 and point2) json files?
\n\n[/quote]
\n\nAbsolutely! I hope they are useful.
", "rawMarkdown": "[quote=Sudeep Juvekar;99439]\r\n\r\nDo we have permission to use the bonnet-tip and blowhead (point1 and point2) json files?\r\n\r\n[/quote]\r\n\r\nAbsolutely! I hope they are useful.", "votes": 1 }, { "id": 102346, "postDate": "2015-12-21T19:27:38.627Z", "content": "Fantastic stuff Anil +1, this is the coolest stuff I've seen in a while :-)
\n\nFor those who ran the code, I am curious about your "heuristic estimate of test error" after 40 epochs (and how this changes with different imwidth, nchan or network depth if you've experimented). I was only able to run with imwidth=384 and nchan=16 due to memory limits (6G) and currently hovering around a test error of 61 (before it goes back higher). As a reference point, each epochs took 140s to run (so ~1.5hr for 40 epochs) and I'd like to experiment in the "right direction" (more epochs vs. smaller crops vs. shallower/deeper network layers).
\n\nPerhaps I should also pre-process the crops further... hmm... anyway not much of a whale enthusiastic myself but I am excited to learn and try the Neon RNN on another Kaggle competition!
\n\nLastly, it was rather painful to get the code running (eventually it worked for me on a fresh ubuntu 14.04 install -- PM me if anyone wants a step-by-step instruction I've put together).
", "rawMarkdown": "Fantastic stuff Anil +1, this is the coolest stuff I've seen in a while :-)\r\n\r\nFor those who ran the code, I am curious about your \"heuristic estimate of test error\" after 40 epochs (and how this changes with different imwidth, nchan or network depth if you've experimented). I was only able to run with imwidth=384 and nchan=16 due to memory limits (6G) and currently hovering around a test error of 61 (before it goes back higher). As a reference point, each epochs took 140s to run (so ~1.5hr for 40 epochs) and I'd like to experiment in the \"right direction\" (more epochs vs. smaller crops vs. shallower/deeper network layers).\r\n\r\nPerhaps I should also pre-process the crops further... hmm... anyway not much of a whale enthusiastic myself but I am excited to learn and try the Neon RNN on another Kaggle competition!\r\n\r\nLastly, it was rather painful to get the code running (eventually it worked for me on a fresh ubuntu 14.04 install -- PM me if anyone wants a step-by-step instruction I've put together).", "votes": 2 }, { "id": 102014, "postDate": "2015-12-18T16:43:40.197Z", "content": "Batch size is usually adjusted to reduce the GPU memory allocation. Try changing 'z32' to 'z8' (or even smaller like z4) in run.sh. Neon, though, does not allow low batch sizes and throws an assertionFailure. A hackish work-around is to find out which line causes assertion failure and commenting it out (#). It's still not guaranteed to work and might change results, but still worth a try.
", "rawMarkdown": "Batch size is usually adjusted to reduce the GPU memory allocation. Try changing 'z32' to 'z8' (or even smaller like z4) in run.sh. Neon, though, does not allow low batch sizes and throws an assertionFailure. A hackish work-around is to find out which line causes assertion failure and commenting it out (#). It's still not guaranteed to work and might change results, but still worth a try.", "votes": 2 }, { "id": 101717, "postDate": "2015-12-16T17:31:33.160Z", "content": "If you are having difficulty with Neon because of older GPU/installation problems, I have translated Anil's models to Caffe. Initial commits are here: https://github.com/sjuvekar/Kaggle-Whales-Caffe.
\n\nThe code is a work-in-progress. I will soon update with an independent post about it.
", "rawMarkdown": "If you are having difficulty with Neon because of older GPU/installation problems, I have translated Anil's models to Caffe. Initial commits are here: https://github.com/sjuvekar/Kaggle-Whales-Caffe.\r\n\r\nThe code is a work-in-progress. I will soon update with an independent post about it.", "votes": 2 }, { "id": 101364, "postDate": "2015-12-15T04:06:14.247Z", "content": "In fact check this github issue: https://github.com/NervanaSystems/neon/issues/80
\n\nThey are probably working on it as we speak.
\n\nGreat Framework! Let's do our best to bump it over Caffe and Theano (and Torch!) here :)
", "rawMarkdown": "In fact check this github issue: https://github.com/NervanaSystems/neon/issues/80\r\n\r\nThey are probably working on it as we speak. \r\n\r\nGreat Framework! Let's do our best to bump it over Caffe and Theano (and Torch!) here :)", "votes": 2 }, { "id": 100448, "postDate": "2015-12-07T16:04:40.717Z", "content": "Hi Julian, your mugshots look great. Certainly worth a better score than 5.4. I was going to say there's probably a bug in your classifier, but it looks like you found it already. Good luck!
", "rawMarkdown": "Hi Julian, your mugshots look great. Certainly worth a better score than 5.4. I was going to say there's probably a bug in your classifier, but it looks like you found it already. Good luck!", "votes": 2 }, { "id": 102208, "postDate": "2015-12-20T12:55:03.037Z", "content": "Traceback (most recent call last):\n File "crop.py", line 98, in \n pool.map(cropfunc, range(pcount))\n File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map\n return self.map_async(func, iterable, chunksize).get()\n File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get\n raise self._value\nTypeError: rotate() got an unexpected keyword argument 'center'\nwhat's the problem with?
", "rawMarkdown": "Traceback (most recent call last):\r\n File \"crop.py\", line 98, in[quote=James King;101964]
\n\n[quote=Anil Thomas;101795]
\n\n[quote]
\n\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon...
\n\n[/quote]
\n\nJames, is this python3? Neon currently needs 2.7.
\n\nThere's some anaconda specific instructions in the docs that may be helpful.
\n\n[/quote]
\n\nI was finally able to get neon to build on a fresh install of Trusty (and confirmed it is using the gpu).
\n\n[/quote]\nIt would be helpful if you can tell how you solved all the problems you had during installation
", "rawMarkdown": "[quote=James King;101964]\r\n\r\n[quote=Anil Thomas;101795]\r\n\r\n[quote]\r\n\r\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon...\r\n\r\n[/quote]\r\n\r\nJames, is this python3? Neon currently needs 2.7.\r\n\r\nThere's some [anaconda specific instructions][1] in the docs that may be helpful.\r\n\r\n\r\n [1]: http://neon.nervanasys.com/docs/latest/user_guide.html#anaconda\r\n\r\n[/quote]\r\n\r\nI was finally able to get neon to build on a fresh install of Trusty (and confirmed it is using the gpu).\r\n\r\n[/quote]\r\nIt would be helpful if you can tell how you solved all the problems you had during installation" }, { "id": 100267, "postDate": "2015-12-04T10:16:56.480Z", "content": "Some codes from the previous human facial key points detection competition can be used on the two points annotations.
\n\n\n\nhttps://github.com/olddocks/caffe-facialkp
", "rawMarkdown": "Some codes from the previous human facial key points detection competition can be used on the two points annotations. \r\n\r\nhttp://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/\r\n\r\nhttps://github.com/olddocks/caffe-facialkp" }, { "id": 101513, "postDate": "2015-12-15T17:36:56.503Z", "content": "Anil, thanks for sharing the code!
\n\nI suppose we cannot run current neon release on AWS GPU instances. Can we have access to Nervana Cloud? Can you suggest other good options.
", "rawMarkdown": "Anil, thanks for sharing the code! \r\n\r\nI suppose we cannot run current neon release on AWS GPU instances. Can we have access to Nervana Cloud? Can you suggest other good options.", "votes": 1 }, { "id": 102436, "postDate": "2015-12-22T17:04:22.567Z", "content": "[quote=James King;102432]
\n\n[quote=yilisg;102430]\nThanks James -- I kept the imwidth and had better luck playing with the nchan and network structure.\n[/quote]\nProbably a good idea. I can get good test crops with lower imwidth but the classifier doesn't run right. Cropping with small enough imwidth to fit in memory and then increasing imwidth back to 384 when classifying seems to work.
\n\nI ran some generic networks in cxxnet to try to classify the neon-produced crops but they did not perform well. I'm very surprised at how well Anil's classifier does with no preprocessing to remove all the irrelevant noise.
\n\n[/quote]\ndelete.
", "rawMarkdown": "[quote=James King;102432]\r\n\r\n[quote=yilisg;102430]\r\nThanks James -- I kept the imwidth and had better luck playing with the nchan and network structure.\r\n[/quote]\r\nProbably a good idea. I can get good test crops with lower imwidth but the classifier doesn't run right. Cropping with small enough imwidth to fit in memory and then increasing imwidth back to 384 when classifying seems to work.\r\n\r\nI ran some generic networks in cxxnet to try to classify the neon-produced crops but they did not perform well. I'm very surprised at how well Anil's classifier does with no preprocessing to remove all the irrelevant noise.\r\n\r\n[/quote]\r\ndelete.", "votes": -1 }, { "id": 102427, "postDate": "2015-12-22T15:50:25.003Z", "content": "Can you push testpoint1.json and testpoint2 in github please?I don't have GPU and need nearly 4000 hours to running the code ,I can't cropped it done untill deadline.
", "rawMarkdown": "Can you push testpoint1.json and testpoint2 in github please?I don't have GPU and need nearly 4000 hours to running the code ,I can't cropped it done untill deadline.", "votes": -2 }, { "id": 119518, "postDate": "2016-05-11T05:26:27.513Z", "content": "can anyone help i extract the head of whale but i dont know how to align the faces of whale please give suggestion how to align .\nthanks \nmouzmeen
", "rawMarkdown": "can anyone help i extract the head of whale but i dont know how to align the faces of whale please give suggestion how to align .\r\nthanks \r\nmouzmeen\r\n" }, { "id": 103811, "postDate": "2016-01-06T18:18:10.797Z", "content": "@anil I am getting this error:
\n\nTraceback (most recent call last):\n File "./classifier.py", line 63, in \n callbacks=callbacks)\n File "/home/ubuntu/wa/1/neon/models/model.py", line 120, in fit\n self._epoch_fit(dataset, callbacks)\n File "/home/ubuntu/wa/1/neon/models/model.py", line 141, in _epoch_fit\n x = self.fprop(x)\n File "/home/ubuntu/wa/1/neon/models/model.py", line 171, in fprop\n return self.layers.fprop(x, inference)\n File "/home/ubuntu/wa/1/neon/layers/container.py", line 103, in fprop\n x = l.fprop(x, inference)\n File "/home/ubuntu/wa/1/neon/layers/layer.py", line 527, in fprop\n self.be.fprop_conv(self.nglayer, inputs, self.W, self.outputs, bsum=self.batch_sum)\n File "/home/ubuntu/wa/whale-2015/neon/backends/nervanacpu.py", line 960, in fprop_conv\n array_O = O.get().reshape(layer.dimO)\nValueError: can only specify one unknown dimension
\n\nThese are the parameters at this point:
\n\nlayer: neon.backends.layer_cpu.ConvLayer object at 0x7f57eb242ad0
\n\nO: CPUTensor(base 0x7f57eb2b4260) name:None shape:(1024, 32) dtype: type 'numpy.float32' strides:(128, 4) is_c_contiguous:True)
\n\ndim: (1024, 1, -1, -1, 32))
", "rawMarkdown": "@anil I am getting this error:\r\n\r\nTraceback (most recent call last):\r\n File \"./classifier.py\", line 63, in[quote=James King;103590]
\n\nIs there any way to make neon print both train and test error with each iteration? The callback method only takes one evaluation set as an argument.
\n\n[/quote]
\n\nDoesn't the progress bar already print out the train error? If you are trying to add a different kind of error metric, maybe define a new Callback class and use the Callbacks.add_callback() function to hook it up.
", "rawMarkdown": "[quote=James King;103590]\r\n\r\nIs there any way to make neon print both train and test error with each iteration? The callback method only takes one evaluation set as an argument.\r\n\r\n[/quote]\r\n\r\nDoesn't the progress bar already print out the train error? If you are trying to add a different kind of error metric, maybe define a new Callback class and use the Callbacks.add_callback() function to hook it up." }, { "id": 103727, "postDate": "2016-01-05T21:07:26.893Z", "content": "[quote=Shize Su;103490]
\n\n@Anil: Hi Anil, it seems that your posted code to generate the test points is not reproducible. There is a seed parameter, but it doesn't seem to work...Would you be so kind to let us know how to make your posted code to generate the test points reproducible?
\n\n[/quote]
\n\nHi Shize, you're right about the results not being exactly reproducible. Your best bet at this point may be to ensemble multiple results to end up with something more stable.
\n\nWe will have a deterministic GPU backend in neon within a couple of weeks.
", "rawMarkdown": "[quote=Shize Su;103490]\r\n\r\n@Anil: Hi Anil, it seems that your posted code to generate the test points is not reproducible. There is a seed parameter, but it doesn't seem to work...Would you be so kind to let us know how to make your posted code to generate the test points reproducible?\r\n\r\n[/quote]\r\n\r\nHi Shize, you're right about the results not being exactly reproducible. Your best bet at this point may be to ensemble multiple results to end up with something more stable.\r\n\r\nWe will have a deterministic GPU backend in neon within a couple of weeks." }, { "id": 103723, "postDate": "2016-01-05T20:47:25.660Z", "content": "[quote=loweew;103449]
\n\nHow straight forward is it to extract features using neon? I have a reasonable model trained, and would like to extract the penultimate layer outputs to use as inputs to a different ML method.
\n\n[/quote]
\n\nYou should be able to directly access the outputs member of the layer that you are interested in. See the function get_outputs() inside evaluator.py for an example of getting the output of the network. In this example, I get the output of the final layer. If you are interested in the penultimate layer, you might want to do something like: model.layers.layers[-2].outputs.get().
", "rawMarkdown": "[quote=loweew;103449]\r\n\r\nHow straight forward is it to extract features using neon? I have a reasonable model trained, and would like to extract the penultimate layer outputs to use as inputs to a different ML method. \r\n\r\n[/quote]\r\n\r\nYou should be able to directly access the *outputs* member of the layer that you are interested in. See the function *get_outputs*() inside [evaluator.py][1] for an example of getting the output of the network. In this example, I get the output of the final layer. If you are interested in the penultimate layer, you might want to do something like: *model.layers.layers[-2].outputs.get().*\r\n\r\n\r\n [1]: https://github.com/anlthms/whale-2015/blob/master/evaluator.py" }, { "id": 103590, "postDate": "2016-01-04T15:56:05.867Z", "content": "Is there any way to make neon print both train and test error with each iteration? The callback method only takes one evaluation set as an argument.
", "rawMarkdown": "Is there any way to make neon print both train and test error with each iteration? The callback method only takes one evaluation set as an argument." }, { "id": 103482, "postDate": "2016-01-03T16:06:28.933Z", "content": "Now tried to change the imwidth=256 , but getting the below error in Classifying:
\n\nFile "/home/roshan/anaconda3/envs/python2/lib/python2.7/site-packages/neon/backends/layer_gpu.py", line 1122, in _magic64\n magic, shift = _magic32(nmax, d)\n File "/home/roshan/anaconda3/envs/python2/lib/python2.7/site-packages/neon/backends/layer_gpu.py", line 1103, in _magic32\n nc = ((nmax + 1) // d) * d - 1\nZeroDivisionError: integer division or modulo by zero
", "rawMarkdown": "Now tried to change the imwidth=256 , but getting the below error in Classifying:\r\n\r\n File \"/home/roshan/anaconda3/envs/python2/lib/python2.7/site-packages/neon/backends/layer_gpu.py\", line 1122, in _magic64\r\n magic, shift = _magic32(nmax, d)\r\n File \"/home/roshan/anaconda3/envs/python2/lib/python2.7/site-packages/neon/backends/layer_gpu.py\", line 1103, in _magic32\r\n nc = ((nmax + 1) // d) * d - 1\r\nZeroDivisionError: integer division or modulo by zero\r\n" }, { "id": 103449, "postDate": "2016-01-03T00:06:31.093Z", "content": "How straight forward is it to extract features using neon? I have a reasonable model trained, and would like to extract the penultimate layer outputs to use as inputs to a different ML method. I couldn't find any working examples or tutorials for this. Any input would be greatly appreciated.
", "rawMarkdown": "How straight forward is it to extract features using neon? I have a reasonable model trained, and would like to extract the penultimate layer outputs to use as inputs to a different ML method. I couldn't find any working examples or tutorials for this. Any input would be greatly appreciated." }, { "id": 103442, "postDate": "2016-01-02T23:02:22.540Z", "content": "lhatsk, if you reduce the imwidth parameter, make sure that the network doesn't squish down the spatial dimensions to invalid values. You can get the network structure printed out by passing -v to classifier.py (this only works for valid networks). Maybe you can unroll the for loop within classifier.py and delete a couple of the maxpooling layers to prevent the dimensions from getting too small as you forward prop the images. Another possibility is setting the strides parameter to 1 in the first layer.
", "rawMarkdown": "lhatsk, if you reduce the imwidth parameter, make sure that the network doesn't squish down the spatial dimensions to invalid values. You can get the network structure printed out by passing ***-v*** to classifier.py (this only works for valid networks). Maybe you can unroll the for loop within [classifier.py][1] and delete a couple of the maxpooling layers to prevent the dimensions from getting too small as you forward prop the images. Another possibility is setting the strides parameter to 1 in the first layer.\r\n\r\n\r\n [1]: https://github.com/anlthms/whale-2015/blob/master/classifier.py" }, { "id": 103245, "postDate": "2015-12-30T16:58:59.123Z", "content": "OK, test error decreases, I get crops but the classifier aborts now. Has anyone an idea? Didn't change anything in the code besides imwidth. Thanks!
\n\n/edit: Post #67 (James King) refers to the same problem and solves it.
\n\nTraceback (most recent call last):
\n\nFile "./classifier.py", line 63, in \n callbacks=callbacks)
\n\nFile "/usr/local/lib/python2.7/dist-packages/neon/models/model.py", line 109, in fit\n self.initialize(dataset, cost)
\n\nFile "/usr/local/lib/python2.7/dist-packages/neon/models/model.py", line 73, in initialize\n prev_input = self.layers.configure(prev_input)
\n\nFile "/usr/local/lib/python2.7/dist-packages/neon/layers/container.py", line 70, in configure\n in_obj = l.configure(in_obj)
\n\nFile "/usr/local/lib/python2.7/dist-packages/neon/layers/layer.py", line 516, in configure\n self.nglayer = self.be.conv_layer(self.be.default_dtype, **self.convparams)
\n\nFile "/usr/local/lib/python2.7/dist-packages/neon/backends/nervanagpu.py", line 1448, in conv_layer\n relu, bsum, deterministic_update)
\n\nFile "/usr/local/lib/python2.7/dist-packages/neon/backends/layer_gpu.py", line 525, in init\n sm_count)
\n\nFile "/usr/local/lib/python2.7/dist-packages/neon/backends/kernel_specs.py", line 549, in update_grid\n return (grid[0][0], grid[0][1], threads)
\n\nIndexError: list index out of range
", "rawMarkdown": "OK, test error decreases, I get crops but the classifier aborts now. Has anyone an idea? Didn't change anything in the code besides imwidth. Thanks!\r\n\r\n/edit: Post #67 (James King) refers to the same problem and solves it. \r\n\r\nTraceback (most recent call last):\r\n\r\n File \"./classifier.py\", line 63, in[quote=yilisg;102346]
\n\nFantastic stuff Anil +1, this is the coolest stuff I've seen in a while :-)
\n\nFor those who ran the code, I am curious about your "heuristic estimate of test error" after 40 epochs (and how this changes with different imwidth, nchan or network depth if you've experimented). I was only able to run with imwidth=384 and nchan=16 due to memory limits (6G) and currently hovering around a test error of 61 (before it goes back higher). As a reference point, each epochs took 140s to run (so ~1.5hr for 40 epochs) and I'd like to experiment in the "right direction" (more epochs vs. smaller crops vs. shallower/deeper network layers).
\n\nPerhaps I should also pre-process the crops further... hmm... anyway not much of a whale enthusiastic myself but I am excited to learn and try the Neon RNN on another Kaggle competition!
\n\nLastly, it was rather painful to get the code running (eventually it worked for me on a fresh ubuntu 14.04 install -- PM me if anyone wants a step-by-step instruction I've put together).
\n\n[/quote]\nHi yilisg, i also installed a fresh ubuntu install in VM-Workstation. my code starts to run but fails around traincrops part. could you please send me your step-by-step instructions for the install ? \nmy email is mpsampat@gmail.com
", "rawMarkdown": "[quote=yilisg;102346]\r\n\r\nFantastic stuff Anil +1, this is the coolest stuff I've seen in a while :-)\r\n\r\nFor those who ran the code, I am curious about your \"heuristic estimate of test error\" after 40 epochs (and how this changes with different imwidth, nchan or network depth if you've experimented). I was only able to run with imwidth=384 and nchan=16 due to memory limits (6G) and currently hovering around a test error of 61 (before it goes back higher). As a reference point, each epochs took 140s to run (so ~1.5hr for 40 epochs) and I'd like to experiment in the \"right direction\" (more epochs vs. smaller crops vs. shallower/deeper network layers).\r\n\r\nPerhaps I should also pre-process the crops further... hmm... anyway not much of a whale enthusiastic myself but I am excited to learn and try the Neon RNN on another Kaggle competition!\r\n\r\nLastly, it was rather painful to get the code running (eventually it worked for me on a fresh ubuntu 14.04 install -- PM me if anyone wants a step-by-step instruction I've put together).\r\n\r\n[/quote]\r\nHi yilisg, i also installed a fresh ubuntu install in VM-Workstation. my code starts to run but fails around traincrops part. could you please send me your step-by-step instructions for the install ? \r\nmy email is mpsampat@gmail.com" }, { "id": 103216, "postDate": "2015-12-30T09:41:37.057Z", "content": "[quote=haiyuansun;102208]
\n\nTraceback (most recent call last):\n File "crop.py", line 98, in \n pool.map(cropfunc, range(pcount))\n File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map\n return self.map_async(func, iterable, chunksize).get()\n File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get\n raise self._value\nTypeError: rotate() got an unexpected keyword argument 'center'\nwhat's the problem with?
\n\n[/quote]\nHi haiyuansun, I have the exact same error. were you able to resolve this issue ? Is yes could you me how you resolved it ?
", "rawMarkdown": "[quote=haiyuansun;102208]\r\n\r\nTraceback (most recent call last):\r\n File \"crop.py\", line 98, inlate to this, but just try to learn how to run deep learning on gpu. my computer has cuda 5.0, but when I try the script, it tells me device 0 does not have cuda compute capability 5.0 or greater. \nRuntimeError: Device 0 does not have CUDA compute capability 5.0 or greater
\n\nmy device config
", "rawMarkdown": "late to this, but just try to learn how to run deep learning on gpu. my computer has cuda 5.0, but when I try the script, it tells me device 0 does not have cuda compute capability 5.0 or greater. \r\nRuntimeError: Device 0 does not have CUDA compute capability 5.0 or greater\r\n\r\n![enter image description here][1]\r\n\r\nmy device config\r\n [1]: http://C:%5CUsers%5Cli%5CDesktop%5Cgraph.png\r\n" }, { "id": 102644, "postDate": "2015-12-24T05:18:39.973Z", "content": "[quote=MrTwiggy;102637]
\n\nIs there support for non-Maxwell GPUs in neon? My main goal is to reproduce the training and test crops to be able to train my own model for identification on cropped images, but installing neon and getting it to work has proven to be insurmountable. I have a Geforce GTX 780 btw, which is Kepler afaik. Has anyone publicly posted a set of JSON points for train/test sets on the bonnet and blowhole for cropping?
\n\n[/quote]
\n\nSee: https://github.com/NervanaSystems/neon/issues/80
\n\nI don't think non-Maxwell support will be ready for this competition.
", "rawMarkdown": "[quote=MrTwiggy;102637]\r\n\r\nIs there support for non-Maxwell GPUs in neon? My main goal is to reproduce the training and test crops to be able to train my own model for identification on cropped images, but installing neon and getting it to work has proven to be insurmountable. I have a Geforce GTX 780 btw, which is Kepler afaik. Has anyone publicly posted a set of JSON points for train/test sets on the bonnet and blowhole for cropping?\r\n\r\n[/quote]\r\n\r\nSee: https://github.com/NervanaSystems/neon/issues/80\r\n\r\nI don't think non-Maxwell support will be ready for this competition." }, { "id": 102637, "postDate": "2015-12-24T03:05:00.740Z", "content": "Is there support for non-Maxwell GPUs in neon? My main goal is to reproduce the training and test crops to be able to train my own model for identification on cropped images, but installing neon and getting it to work has proven to be insurmountable. I have a Geforce GTX 780 btw, which is Kepler afaik. Has anyone publicly posted a set of JSON points for train/test sets on the bonnet and blowhole for cropping?
", "rawMarkdown": "Is there support for non-Maxwell GPUs in neon? My main goal is to reproduce the training and test crops to be able to train my own model for identification on cropped images, but installing neon and getting it to work has proven to be insurmountable. I have a Geforce GTX 780 btw, which is Kepler afaik. Has anyone publicly posted a set of JSON points for train/test sets on the bonnet and blowhole for cropping?" }, { "id": 102458, "postDate": "2015-12-22T19:37:29.380Z", "content": "[quote=jwjohnson314;102454]
\n\n[quote=Sudeep Juvekar;101717]
\n\nIf you are having difficulty with Neon because of older GPU/installation problems, I have translated Anil's models to Caffe. Initial commits are here: https://github.com/sjuvekar/Kaggle-Whales-Caffe.
\n\nThe code is a work-in-progress. I will soon update with an independent post about it.
\n\n[/quote]
\n\nSudeep, have you had success running the Caffe code? I run out of memory on a K40 with 12gb, even after removing layers from the model.
\n\n[/quote]
\n\nClassifier runs fine with cropped training images (classifier_solver.prototxt in latest commit in solver/ or last line in run.sh). Cropping code is not complete yet, some people have volunteered to modify it.
", "rawMarkdown": "[quote=jwjohnson314;102454]\r\n\r\n[quote=Sudeep Juvekar;101717]\r\n\r\nIf you are having difficulty with Neon because of older GPU/installation problems, I have translated Anil's models to Caffe. Initial commits are here: https://github.com/sjuvekar/Kaggle-Whales-Caffe.\r\n\r\nThe code is a work-in-progress. I will soon update with an independent post about it.\r\n\r\n[/quote]\r\n\r\nSudeep, have you had success running the Caffe code? I run out of memory on a K40 with 12gb, even after removing layers from the model.\r\n\r\n[/quote]\r\n\r\nClassifier runs fine with cropped training images (classifier_solver.prototxt in latest commit in solver/ or last line in run.sh). Cropping code is not complete yet, some people have volunteered to modify it." }, { "id": 102454, "postDate": "2015-12-22T18:53:22.823Z", "content": "[quote=Sudeep Juvekar;101717]
\n\nIf you are having difficulty with Neon because of older GPU/installation problems, I have translated Anil's models to Caffe. Initial commits are here: https://github.com/sjuvekar/Kaggle-Whales-Caffe.
\n\nThe code is a work-in-progress. I will soon update with an independent post about it.
\n\n[/quote]
\n\nSudeep, have you had success running the Caffe code? I run out of memory on a K40 with 12gb, even after removing layers from the model.
", "rawMarkdown": "[quote=Sudeep Juvekar;101717]\r\n\r\nIf you are having difficulty with Neon because of older GPU/installation problems, I have translated Anil's models to Caffe. Initial commits are here: https://github.com/sjuvekar/Kaggle-Whales-Caffe.\r\n\r\nThe code is a work-in-progress. I will soon update with an independent post about it.\r\n\r\n[/quote]\r\n\r\nSudeep, have you had success running the Caffe code? I run out of memory on a K40 with 12gb, even after removing layers from the model." }, { "id": 102430, "postDate": "2015-12-22T16:33:50.863Z", "content": "Thanks James -- I kept the imwidth and had better luck playing with the nchan and network structure.
\n\n@Anil Xmas came early and I've just added a second GPU to my machine. On other software, I was able to use the entire 12gb (2x6gb), but in neon I am still getting the error (cuMemAlloc failed: out of memory). I understand since v0.9 neon supports multiple GPUs -- but after I enable -bmgpu I get the following error ("No module named mgpu.nervanamgpu"). Kindly let me know what should I do to enable two GPUs (perhaps a special kaggler release where max_devices is capped at 2?). Thanks!
\n\nIn the meantime, I suppose I could run two processes in parallel, each assigned to a different GPU but that wouldn't solve my ram limitation :(
", "rawMarkdown": "Thanks James -- I kept the imwidth and had better luck playing with the nchan and network structure.\r\n\r\n@Anil Xmas came early and I've just added a second GPU to my machine. On other software, I was able to use the entire 12gb (2x6gb), but in neon I am still getting the error (cuMemAlloc failed: out of memory). I understand since v0.9 neon supports multiple GPUs -- but after I enable -bmgpu I get the following error (\"No module named mgpu.nervanamgpu\"). Kindly let me know what should I do to enable two GPUs (perhaps a special kaggler release where max_devices is capped at 2?). Thanks!\r\n\r\nIn the meantime, I suppose I could run two processes in parallel, each assigned to a different GPU but that wouldn't solve my ram limitation :(" }, { "id": 102351, "postDate": "2015-12-21T20:32:02.193Z", "content": ""Heuristic estimate of test error" bounced around between 30 and 40 using imwidth = 224, no change to nchan or network topology. The localizer for point 2 stopped early, I forget which iteration. The test crops looked pretty good but still only got 5.51 on the leader board.
", "rawMarkdown": "\"Heuristic estimate of test error\" bounced around between 30 and 40 using imwidth = 224, no change to nchan or network topology. The localizer for point 2 stopped early, I forget which iteration. The test crops looked pretty good but still only got 5.51 on the leader board." }, { "id": 102235, "postDate": "2015-12-20T20:02:12.887Z", "content": "[quote=James King;102233]
\n\nWell, I was able to produce some reasonable test crops with imwidth=192, but the classifier crashes when it passes P = -1 and Q = -1 to update_grid in kernel_specs.py.
\n\n[/quote]
\n\nI was able to fix the crash by changing for idx in range(6): \nto \nfor idx in range(5):
but only got an lb score of 6.36. Maybe time to try a different approach.
", "rawMarkdown": "[quote=James King;102233]\r\n\r\nWell, I was able to produce some reasonable test crops with imwidth=192, but the classifier crashes when it passes P = -1 and Q = -1 to update_grid in kernel_specs.py.\r\n\r\n[/quote]\r\n\r\nI was able to fix the crash by changing `for idx in range(6):` \r\nto \r\n`for idx in range(5):`\r\n\r\nbut only got an lb score of 6.36. Maybe time to try a different approach." }, { "id": 102233, "postDate": "2015-12-20T19:32:11.683Z", "content": "Well, I was able to produce some reasonable test crops with imwidth=192, but the classifier crashes when it passes P = -1 and Q = -1 to update_grid in kernel_specs.py.
", "rawMarkdown": "Well, I was able to produce some reasonable test crops with imwidth=192, but the classifier crashes when it passes P = -1 and Q = -1 to update_grid in kernel_specs.py." }, { "id": 102182, "postDate": "2015-12-20T08:35:46.850Z", "content": "[quote=Anil Thomas;102163]
\n\nMaking the network less deep is slightly trickier. There are two loops in localizer.py that adds conv and deconv layers. Currently they are hardcoded to execute 16 and 15 times respectively. You can try changing those numbers to something like 8 and 7, for example.
\n\n[/quote]
\n\nThanks Anil, this helps. Considering that the localizer networks are not learning any high level/abstract representations, how important is depth in this case?
\n\nIt will be interesting to find the shallowest network capable of learning bonnet/blowhole points, worth running a few experiments.
", "rawMarkdown": "[quote=Anil Thomas;102163]\r\n\r\nMaking the network less deep is slightly trickier. There are two loops in [localizer.py][2] that adds conv and deconv layers. Currently they are hardcoded to execute 16 and 15 times respectively. You can try changing those numbers to something like 8 and 7, for example.\r\n\r\n [2]: https://github.com/anlthms/whale-2015/blob/master/localizer.py\r\n\r\n[/quote]\r\n\r\nThanks Anil, this helps. Considering that the localizer networks are not learning any high level/abstract representations, how important is depth in this case?\r\n\r\nIt will be interesting to find the shallowest network capable of learning bonnet/blowhole points, worth running a few experiments." }, { "id": 102165, "postDate": "2015-12-20T03:57:30.367Z", "content": "[quote]
\n\n@lhatsk, I too had the same issues with loader folder , and after that faced cuMemAlloc failed issue. Changed 'z32' to 'z8' as Sudeep suggested (with 4GB Gpu) and its running fine now, but slow...currently in 9th Epoch and it took around 20 hours :(
\n\n[/quote]
\n\n@RKR, that sounds like the code is running on the CPU. Try running nvidia-smi to see if the GPU is being utilized at all.
[quote]
\n\nBatch size is usually adjusted to reduce the GPU memory allocation. Try changing 'z32' to 'z8' (or even smaller like z4) in run.sh. Neon, though, does not allow low batch sizes and throws an assertionFailure. A hackish work-around is to find out which line causes assertion failure and commenting it out (#). It's still not guaranteed to work and might change results, but still worth a try.
\n\n[/quote]
\n\nI wouldn't recommend that. The assertion is present in the code for a good reason. If you are running this on a CPU, you can set the batch size to whatever you like, but on a GPU the options are limited.
", "rawMarkdown": "[quote]\r\n\r\nBatch size is usually adjusted to reduce the GPU memory allocation. Try changing 'z32' to 'z8' (or even smaller like z4) in run.sh. Neon, though, does not allow low batch sizes and throws an assertionFailure. A hackish work-around is to find out which line causes assertion failure and commenting it out (#). It's still not guaranteed to work and might change results, but still worth a try.\r\n\r\n[/quote]\r\n\r\nI wouldn't recommend that. The assertion is present in the code for a good reason. If you are running this on a CPU, you can set the batch size to whatever you like, but on a GPU the options are limited.\r\n" }, { "id": 102162, "postDate": "2015-12-20T03:25:14.977Z", "content": "[quote]
\n\nThanks for sharing Anil. Great code and the Neon framework looks neat.\n I am experimenting with your encoder and I was wondering if in the LocalizerLoader there is a quicker way to set the mask other than passing floats one by one to the rather slow function call_compound_kernel() ?
\n\n[/quote]
\n\nGreat to know you are having a good experience with neon.
\n\nInstead of setting each pixel individually, you could make the target circle in advance. Keep that image in GPU memory and copy it to the appropriate location within the mask as needed.
\n\nMost of the compute time is spent training the network, so you are unlikely to save more than a few minutes by optimizing the data loading part.
", "rawMarkdown": "[quote]\r\n\r\nThanks for sharing Anil. Great code and the Neon framework looks neat.\r\n I am experimenting with your encoder and I was wondering if in the LocalizerLoader there is a quicker way to set the mask other than passing floats one by one to the rather slow function call_compound_kernel() ?\r\n\r\n[/quote]\r\n\r\nGreat to know you are having a good experience with neon.\r\n\r\nInstead of setting each pixel individually, you could make the target circle in advance. Keep that image in GPU memory and copy it to the appropriate location within the mask as needed.\r\n\r\nMost of the compute time is spent training the network, so you are unlikely to save more than a few minutes by optimizing the data loading part." }, { "id": 102129, "postDate": "2015-12-19T18:26:23.070Z", "content": "its around 12.14 , but its taking 4900 seconds for each epoch :(
", "rawMarkdown": "its around 12.14 , but its taking 4900 seconds for each epoch :(" }, { "id": 102122, "postDate": "2015-12-19T16:37:47.723Z", "content": "@RKR Yes, changing it to 'z8' works. I just ran two epochs, each took around 900s. Any idea how much impact the batch size change has on the performance? The first errors were around 12.7...
", "rawMarkdown": "@RKR Yes, changing it to 'z8' works. I just ran two epochs, each took around 900s. Any idea how much impact the batch size change has on the performance? The first errors were around 12.7..." }, { "id": 102118, "postDate": "2015-12-19T16:26:32.593Z", "content": "@lhatsk, I too had the same issues with loader folder , and after that faced cuMemAlloc failed issue. Changed 'z32' to 'z8' as Sudeep suggested (with 4GB Gpu) and its running fine now, but slow...currently in 9th Epoch and it took around 20 hours :(
", "rawMarkdown": "@lhatsk, I too had the same issues with loader folder , and after that faced cuMemAlloc failed issue. Changed 'z32' to 'z8' as Sudeep suggested (with 4GB Gpu) and its running fine now, but slow...currently in 9th Epoch and it took around 20 hours :(" }, { "id": 102111, "postDate": "2015-12-19T14:59:45.237Z", "content": "I had the same problem with loader.so, it looks like everything was build during make sysinstall, but the loader directory isn't copied to /usr/local/lib/python2.7/dist-packages/neon/data in my case. So I created a loader directory in data and copied over loader.so. I'm now at the next stage with cuMemAlloc failed: out of memory :-)
", "rawMarkdown": "I had the same problem with loader.so, it looks like everything was build during make sysinstall, but the loader directory isn't copied to /usr/local/lib/python2.7/dist-packages/neon/data in my case. So I created a loader directory in data and copied over loader.so. I'm now at the next stage with cuMemAlloc failed: out of memory :-)" }, { "id": 102075, "postDate": "2015-12-19T04:27:56.223Z", "content": "The progress bar in this code is top notch :)
", "rawMarkdown": "The progress bar in this code is top notch :)" }, { "id": 102021, "postDate": "2015-12-18T17:40:18.663Z", "content": "I reduced nchan to the point where the out of memory error went away, but then got an nvcc compile error
kernel.cu(26): error: identifier "None" is undefined\nbecause the code in question contains the statement
\n\n#define THREADS None\nUPDATE: Batch size reduction as suggested by Sudeep above looks promising - it's training and through 53 batches.
", "rawMarkdown": "I reduced `nchan` to the point where the out of memory error went away, but then got an nvcc compile error\r\n\r\n kernel.cu(26): error: identifier \"None\" is undefined\r\n\r\nbecause the code in question contains the statement\r\n\r\n #define THREADS None\r\n\r\n***UPDATE: Batch size reduction as suggested by Sudeep above looks promising - it's training and through 53 batches.***\r\n" }, { "id": 102009, "postDate": "2015-12-18T15:10:18.527Z", "content": "@Anil is there a way to fit the localizer using less memory? It exhausts the memory on my 4GB gpu:
\n\n\nTraceback (most recent call last):\n File "./localizer.py", line 77, in \n callbacks=callbacks)\n File "/home/jfk/neon/neon/models/model.py", line 109, in fit\n self.initialize(dataset, cost)\n File "/home/jfk/neon/neon/models/model.py", line 79, in initialize\n self.layers.allocate()\n File "/home/jfk/neon/neon/layers/container.py", line 82, in allocate\n l.allocate()\n File "/home/jfk/neon/neon/layers/layer.py", line 1201, in allocate\n super(BatchNorm, self).allocate(shared_outputs)\n File "/home/jfk/neon/neon/layers/layer.py", line 123, in allocate\n parallelism=self.parallelism)\n File "/home/jfk/neon/neon/backends/backend.py", line 516, in iobuf\n persist_values=persist_values)\n File "/home/jfk/neon/neon/backends/nervanagpu.py", line 1026, in zeros\n rounding=self.round_mode)._assign(0)\n File "/home/jfk/neon/neon/backends/nervanagpu.py", line 132, in init\n self.gpudata = allocator(self.nbytes)\npycuda._driver.MemoryError: cuMemAlloc failed: out of memory\n
", "rawMarkdown": "@Anil is there a way to fit the localizer using less memory? It exhausts the memory on my 4GB gpu:\r\n\r\n\r\nTraceback (most recent call last):\r\n File \"./localizer.py\", line 77, in\r\n callbacks=callbacks)\r\n File \"/home/jfk/neon/neon/models/model.py\", line 109, in fit\r\n self.initialize(dataset, cost)\r\n File \"/home/jfk/neon/neon/models/model.py\", line 79, in initialize\r\n self.layers.allocate()\r\n File \"/home/jfk/neon/neon/layers/container.py\", line 82, in allocate\r\n l.allocate()\r\n File \"/home/jfk/neon/neon/layers/layer.py\", line 1201, in allocate\r\n super(BatchNorm, self).allocate(shared_outputs)\r\n File \"/home/jfk/neon/neon/layers/layer.py\", line 123, in allocate\r\n parallelism=self.parallelism)\r\n File \"/home/jfk/neon/neon/backends/backend.py\", line 516, in iobuf\r\n persist_values=persist_values)\r\n File \"/home/jfk/neon/neon/backends/nervanagpu.py\", line 1026, in zeros\r\n rounding=self.round_mode)._assign(0)\r\n File \"/home/jfk/neon/neon/backends/nervanagpu.py\", line 132, in __init__\r\n self.gpudata = allocator(self.nbytes)\r\npycuda._driver.MemoryError: cuMemAlloc failed: out of memory\r\n \r\n" }, { "id": 101968, "postDate": "2015-12-18T05:25:15.527Z", "content": "[quote=James King;101967]
\n\n[quote=DataGeek;101965]\nIt would be helpful if you can tell how you solved all the problems you had during installation\n[/quote]
\n\nThere's not that much to say. After many futile efforts with my existing environment (couldn't find numpy, numpy was the wrong version, pycuda would not install correctly) I decided to create a fresh environment in a VM. This gave me a working install of neon, but then I realized the VM could not see the gpu. So I loaded Ubuntu trusty onto a usb and created a new partition with a completely clean install. Blacklisted nouveau, installed nvidia drivers, and installed cuda per NVIDIA's instructions
\n\nhttp://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/#axzz3ue67q5TI
\n\nThen I followed the instructions on github for installing neon (in a venv, not the sysinstall) and it worked on the first try.
\n\n[/quote]
\n\nThanks. I think I also need to do the same.
", "rawMarkdown": "[quote=James King;101967]\r\n\r\n[quote=DataGeek;101965]\r\nIt would be helpful if you can tell how you solved all the problems you had during installation\r\n[/quote]\r\n\r\nThere's not that much to say. After many futile efforts with my existing environment (couldn't find numpy, numpy was the wrong version, pycuda would not install correctly) I decided to create a fresh environment in a VM. This gave me a working install of neon, but then I realized the VM could not see the gpu. So I loaded Ubuntu trusty onto a usb and created a new partition with a completely clean install. Blacklisted nouveau, installed nvidia drivers, and installed cuda per NVIDIA's instructions \r\n\r\nhttp://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/#axzz3ue67q5TI\r\n\r\nThen I followed the instructions on github for installing neon (in a venv, not the sysinstall) and it worked on the first try.\r\n\r\n\r\n[/quote]\r\n\r\nThanks. I think I also need to do the same." }, { "id": 101966, "postDate": "2015-12-18T04:54:42.397Z", "content": "[quote=Jesse;101963]
\n\nI'm still unsure of what the input shapes are and if there are any augmentations being used in the neon version, but for anyone who's curious I'm able to get it to learn better (~3.4 on my holdout) if I use batch normalization and adadelta on a smaller network (1/4th the number of filters in each layer). Before I was using Nesterov SGD, so it seems batch norm and adadelta are necessary to some extent.
\n\n[quote=Jesse;101657]
\n\nCould someone explain what the input to the classifier looks like (shape, etc.)? The train crops ...
\n\n[/quote]
\n\n[/quote]\nIt would be nice if you can post some code :)
", "rawMarkdown": "[quote=Jesse;101963]\r\n\r\nI'm still unsure of what the input shapes are and if there are any augmentations being used in the neon version, but for anyone who's curious I'm able to get it to learn better (~3.4 on my holdout) if I use batch normalization and adadelta on a smaller network (1/4th the number of filters in each layer). Before I was using Nesterov SGD, so it seems batch norm and adadelta are necessary to some extent.\r\n\r\n[quote=Jesse;101657]\r\n\r\nCould someone explain what the input to the classifier looks like (shape, etc.)? The train crops ...\r\n\r\n[/quote]\r\n\r\n\r\n[/quote]\r\nIt would be nice if you can post some code :)" }, { "id": 101964, "postDate": "2015-12-18T04:42:49.750Z", "content": "[quote=Anil Thomas;101795]
\n\n[quote]
\n\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon...
\n\n[/quote]
\n\nJames, is this python3? Neon currently needs 2.7.
\n\nThere's some anaconda specific instructions in the docs that may be helpful.
\n\n[/quote]
\n\nI was finally able to get neon to build on a fresh install of Trusty (and confirmed it is using the gpu).
", "rawMarkdown": "[quote=Anil Thomas;101795]\r\n\r\n[quote]\r\n\r\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon...\r\n\r\n[/quote]\r\n\r\nJames, is this python3? Neon currently needs 2.7.\r\n\r\nThere's some [anaconda specific instructions][1] in the docs that may be helpful.\r\n\r\n\r\n [1]: http://neon.nervanasys.com/docs/latest/user_guide.html#anaconda\r\n\r\n[/quote]\r\n\r\nI was finally able to get neon to build on a fresh install of Trusty (and confirmed it is using the gpu)." }, { "id": 101936, "postDate": "2015-12-17T23:44:16.920Z", "content": "[quote=Anil Thomas;101795]
\n\n[quote]
\n\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon:
\n\n\n\nRuntimeError: module compiled against API version a but this version of numpy is 9\nterminate called after throwing an instance of 'std::runtime_error'\n what(): numpy failed to initialize\nAborted (core dumped)\nMakefile:151: recipe for target 'kernels' failed\nmake: *** [kernels] Error 134\nThis is Ubuntu 15.10
\n\n[/quote]
\n\nJames, is this python3? Neon currently needs 2.7.
\n\nThere's some anaconda specific instructions in the docs that may be helpful.
\n\n[/quote]
\n\nIt's python 2.7. make sysinstall fails with the same error; this is using /usr/bin/python2.7 (trying it without anaconda).
", "rawMarkdown": "[quote=Anil Thomas;101795]\r\n\r\n[quote]\r\n\r\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon:\r\n\r\n RuntimeError: module compiled against API version a but this version of numpy is 9\r\n terminate called after throwing an instance of 'std::runtime_error'\r\n what(): numpy failed to initialize\r\n Aborted (core dumped)\r\n Makefile:151: recipe for target 'kernels' failed\r\n make: *** [kernels] Error 134\r\n\r\nThis is Ubuntu 15.10\r\n\r\n[/quote]\r\n\r\nJames, is this python3? Neon currently needs 2.7.\r\n\r\nThere's some [anaconda specific instructions][1] in the docs that may be helpful.\r\n\r\n\r\n [1]: http://neon.nervanasys.com/docs/latest/user_guide.html#anaconda\r\n\r\n[/quote]\r\n\r\nIt's python 2.7. make sysinstall fails with the same error; this is using /usr/bin/python2.7 (trying it without anaconda).\r\n" }, { "id": 101795, "postDate": "2015-12-17T04:30:42.417Z", "content": "[quote]
\n\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon:
\n\n\n\nRuntimeError: module compiled against API version a but this version of numpy is 9\nterminate called after throwing an instance of 'std::runtime_error'\n what(): numpy failed to initialize\nAborted (core dumped)\nMakefile:151: recipe for target 'kernels' failed\nmake: *** [kernels] Error 134\nThis is Ubuntu 15.10
\n\n[/quote]
\n\nJames, is this python3? Neon currently needs 2.7.
\n\nThere's some anaconda specific instructions in the docs that may be helpful.
", "rawMarkdown": "[quote]\r\n\r\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon:\r\n\r\n RuntimeError: module compiled against API version a but this version of numpy is 9\r\n terminate called after throwing an instance of 'std::runtime_error'\r\n what(): numpy failed to initialize\r\n Aborted (core dumped)\r\n Makefile:151: recipe for target 'kernels' failed\r\n make: *** [kernels] Error 134\r\n\r\nThis is Ubuntu 15.10\r\n\r\n[/quote]\r\n\r\nJames, is this python3? Neon currently needs 2.7.\r\n\r\nThere's some [anaconda specific instructions][1] in the docs that may be helpful.\r\n\r\n\r\n [1]: http://neon.nervanasys.com/docs/latest/user_guide.html#anaconda" }, { "id": 101788, "postDate": "2015-12-17T03:30:06.917Z", "content": "[quote=Anil Thomas;101381]
\n\n[quote]
\n\nAnil, is it correct that Neon cannot be installed if running Anaconda due to the use of a virtualenv?
\n\n[/quote]
\n\nLet me know if you are seeing a specific issue. If virtualenv doesn't work, there's also the system-wide install option:
\n\n\n\ncd neon\nmake sysinstall \n[/quote]
\n\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon:
\n\n\n\nRuntimeError: module compiled against API version a but this version of numpy is 9\nterminate called after throwing an instance of 'std::runtime_error'\n what(): numpy failed to initialize\nAborted (core dumped)\nMakefile:151: recipe for target 'kernels' failed\nmake: *** [kernels] Error 134\nThis is Ubuntu 15.10
", "rawMarkdown": "[quote=Anil Thomas;101381]\r\n\r\n[quote]\r\n\r\nAnil, is it correct that Neon cannot be installed if running Anaconda due to the use of a virtualenv?\r\n\r\n[/quote]\r\n\r\nLet me know if you are seeing a specific issue. If virtualenv doesn't work, there's also the system-wide install option:\r\n\r\n cd neon\r\n make sysinstall \r\n\r\n[/quote]\r\n\r\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon:\r\n\r\n RuntimeError: module compiled against API version a but this version of numpy is 9\r\n terminate called after throwing an instance of 'std::runtime_error'\r\n what(): numpy failed to initialize\r\n Aborted (core dumped)\r\n Makefile:151: recipe for target 'kernels' failed\r\n make: *** [kernels] Error 134\r\n\r\nThis is Ubuntu 15.10" }, { "id": 101690, "postDate": "2015-12-16T14:14:32.997Z", "content": "@Anil, Even i got the same error "AttributeError: 'LocalizerLoader' . There were no issues while sys-wide install (make sysinstall) and i did a 'make' in data/loadeer folder as well
", "rawMarkdown": "@Anil, Even i got the same error \"AttributeError: 'LocalizerLoader' . There were no issues while sys-wide install (make sysinstall) and i did a 'make' in data/loadeer folder as well" }, { "id": 101652, "postDate": "2015-12-16T09:21:09.790Z", "content": "Abhishek could you illustrate a bit on the problem regarding "AttributeError: 'LocalizerLoader' object has no attribute 'loaderlib'" ?
\n\nI went into the same problem too, and I don't have nvcc on my server.
", "rawMarkdown": "Abhishek could you illustrate a bit on the problem regarding \"AttributeError: 'LocalizerLoader' object has no attribute 'loaderlib'\" ?\r\n\r\nI went into the same problem too, and I don't have nvcc on my server. " }, { "id": 101385, "postDate": "2015-12-15T05:58:37.663Z", "content": "I think make got interrupted at some point. I have solved this issue :) Thanks!
", "rawMarkdown": "I think make got interrupted at some point. I have solved this issue :) Thanks!" }, { "id": 101384, "postDate": "2015-12-15T05:56:37.543Z", "content": "[quote]
\n\nDid you also get this error:
\n\n\n\n\n2015-12-15 04:04:03,589 - neon.data.imageloader - ERROR - Unable to\n load loader.so.
\n[/quote]
\n\nAbhishek, if you have nvcc in the path, make will attempt to build loader.so using nvcc. If not, it will try gcc (in the latter case, you won't be able to use any GPU features).
\n\nGive this a try to narrow down the issue:
\n\n", "rawMarkdown": "[quote]\r\n\r\nDid you also get this error:\r\n\r\n> 2015-12-15 04:04:03,589 - neon.data.imageloader - ERROR - Unable to\r\n> load loader.so.\r\n\r\n[/quote]\r\n\r\nAbhishek, if you have nvcc in the path, *make* will attempt to build loader.so using nvcc. If not, it will try gcc (in the latter case, you won't be able to use any GPU features).\r\n\r\nGive this a try to narrow down the issue:\r\n\r\n cd neon/data/loader\r\n make\r\n" }, { "id": 101381, "postDate": "2015-12-15T05:47:52.960Z", "content": "cd neon/data/loader\nmake\n[quote]
\n\nAnil, is it correct that Neon cannot be installed if running Anaconda due to the use of a virtualenv?
\n\n[/quote]
\n\nLet me know if you are seeing a specific issue. If virtualenv doesn't work, there's also the system-wide install option:
\n\n", "rawMarkdown": "[quote]\r\n\r\nAnil, is it correct that Neon cannot be installed if running Anaconda due to the use of a virtualenv?\r\n\r\n[/quote]\r\n\r\nLet me know if you are seeing a specific issue. If virtualenv doesn't work, there's also the system-wide install option:\r\n\r\n cd neon\r\n make sysinstall " }, { "id": 101375, "postDate": "2015-12-15T04:56:18.923Z", "content": "cd neon\nmake sysinstall \n[quote=Sudeep Juvekar;101368]
\n\nThat shouldn't happen. Did your 'make' run successfully? It should create .so in neon/data/loader/. Did you activate venv in neon?
\n\nProblem with me is that the cuda kernel fails to compile with Kepler in localize.py
\n\nIt seems like they used to have a support for older GPUs in an earlier version (0.9.0), but it was deprecated. Try running 'make -e GPU=cudanet' (I doubt it will work with latest commits though). Just in case it succeeds, please report back :D
\n\n[/quote]
\n\nNo. It doesnt. On CPU, it will take 15 hours, lol ( for one epoch :( )
", "rawMarkdown": "[quote=Sudeep Juvekar;101368]\r\n\r\nThat shouldn't happen. Did your 'make' run successfully? It should create .so in neon/data/loader/. Did you activate venv in neon?\r\n\r\nProblem with me is that the cuda kernel fails to compile with Kepler in localize.py\r\n\r\nIt seems like they used to have a support for older GPUs in an earlier version (0.9.0), but it was deprecated. Try running 'make -e GPU=cudanet' (I doubt it will work with latest commits though). Just in case it succeeds, please report back :D\r\n\r\n[/quote]\r\n\r\nNo. It doesnt. On CPU, it will take 15 hours, lol ( for one epoch :( )\r\n" }, { "id": 101368, "postDate": "2015-12-15T04:21:13.657Z", "content": "That shouldn't happen. Did your 'make' run successfully? It should create .so in neon/data/loader/. Did you activate venv in neon?
\n\nProblem with me is that the cuda kernel fails to compile with Kepler in localize.py
\n\nIt seems like they used to have a support for older GPUs in an earlier version (0.9.0), but it was deprecated. Try running 'make -e GPU=cudanet' (I doubt it will work with latest commits though). Just in case it succeeds, please report back :D
", "rawMarkdown": "That shouldn't happen. Did your 'make' run successfully? It should create .so in neon/data/loader/. Did you activate venv in neon?\r\n\r\nProblem with me is that the cuda kernel fails to compile with Kepler in localize.py\r\n\r\nIt seems like they used to have a support for older GPUs in an earlier version (0.9.0), but it was deprecated. Try running 'make -e GPU=cudanet' (I doubt it will work with latest commits though). Just in case it succeeds, please report back :D" }, { "id": 101347, "postDate": "2015-12-15T02:56:33.100Z", "content": "[quote=James King;101344]
\n\nAnil, is it correct that Neon cannot be installed if running Anaconda due to the use of a virtualenv?
\n\n[/quote]
\n\nI don't think so. Go to folder anaconda -> lib -> python-2.7 -> site-packages -> git clone https://github.com/NervanaSystems/neon
\n\ncd neon
\n\npython setupy.py install
\n\nwill be installed in few seconds
\n\n*I might be wrong but I have used the same method to install neon and used in other competition before. They might have made some changes in neon repo.
", "rawMarkdown": "[quote=James King;101344]\r\n\r\nAnil, is it correct that Neon cannot be installed if running Anaconda due to the use of a virtualenv?\r\n\r\n[/quote]\r\n\r\nI don't think so. Go to folder anaconda -> lib -> python-2.7 -> site-packages -> git clone https://github.com/NervanaSystems/neon\r\n\r\ncd neon \r\n\r\npython setupy.py install\r\n\r\nwill be installed in few seconds\r\n\r\n*I might be wrong but I have used the same method to install neon and used in other competition before. They might have made some changes in neon repo." }, { "id": 101344, "postDate": "2015-12-15T02:32:18.027Z", "content": "Anil, is it correct that Neon cannot be installed if running Anaconda due to the use of a virtualenv?
", "rawMarkdown": "Anil, is it correct that Neon cannot be installed if running Anaconda due to the use of a virtualenv?" }, { "id": 101204, "postDate": "2015-12-14T18:58:45.717Z", "content": "[quote]
\n\nI see actually at the first position of the leaderboard there is another guy also from Nervana, do you use similar approaches or his method is not related to yours?
\n\n[/quote]
\n\nHe used the exact same code that is published on github (with the bug and all). Dr. Luke is just luckier than you and me ;-)
", "rawMarkdown": "[quote]\r\n\r\nI see actually at the first position of the leaderboard there is another guy also from Nervana, do you use similar approaches or his method is not related to yours?\r\n\r\n[/quote]\r\n\r\nHe used the exact same code that is published on github (with the bug and all). Dr. Luke is just luckier than you and me ;-)\r\n" }, { "id": 101165, "postDate": "2015-12-14T13:58:57.473Z", "content": "[quote=Anil Thomas;101115]
\n\nHold on! Found a bug in the code. It turns out that the probability of getting a 3.5ish score in a single try is less than 0.25. Will push an update tomorrow that should make it more reliable.
\n\n[/quote]
\n\nThank you for sharing, your code looks neat, waiting for your bug fix. We just ran into this competiton, so your work could be a good start point.
\n\nI see actually at the first position of the leaderboard there is another guy also from Nervana, do you use similar approaches or his method is not related to yours?
", "rawMarkdown": "[quote=Anil Thomas;101115]\r\n\r\nHold on! Found a bug in the code. It turns out that the probability of getting a 3.5ish score in a single try is less than 0.25. Will push an update tomorrow that should make it more reliable.\r\n\r\n[/quote]\r\n\r\n\r\nThank you for sharing, your code looks neat, waiting for your bug fix. We just ran into this competiton, so your work could be a good start point.\r\n\r\nI see actually at the first position of the leaderboard there is another guy also from Nervana, do you use similar approaches or his method is not related to yours?" }, { "id": 101140, "postDate": "2015-12-14T09:19:20.243Z", "content": "righto, I ran the code twice. On both occasions, the encoder stopped early at ~10 epochs, and produced pretty bad crops, probably 5% of the crops look reasonable. The classifier produced a test score of ~6.6.
\n\n[quote=Anil Thomas;101115]
\n\nHold on! Found a bug in the code. It turns out that the probability of getting a 3.5ish score in a single try is less than 0.25. Will push an update tomorrow that should make it more reliable.
\n\n[/quote]
", "rawMarkdown": "righto, I ran the code twice. On both occasions, the encoder stopped early at ~10 epochs, and produced pretty bad crops, probably 5% of the crops look reasonable. The classifier produced a test score of ~6.6.\r\n\r\n[quote=Anil Thomas;101115]\r\n\r\nHold on! Found a bug in the code. It turns out that the probability of getting a 3.5ish score in a single try is less than 0.25. Will push an update tomorrow that should make it more reliable.\r\n\r\n[/quote]\r\n" }, { "id": 100913, "postDate": "2015-12-11T23:41:02.193Z", "content": "Cool, now everyone's using Neon for this. Such an effective way to gain attention in a crowded space of deep learning tools...
\n\nbut I think it's well worth it. Thanks for the generous sharing! Will be the 1st time to try Neon, but it seems cool.
\n\n[quote=Anil Thomas;100902]
\n\nI have been threatening to release the source code for a while. Here it is, finally! With 4 weeks left before the deadline, there's hopefully enough time for everyone to process the code and make improvements.
\n\nAs given, it should reproduce my current score (which is at second place on the leaderboard). Note that you will see quite a bit of variation from run to run, so it could take multiple tries to get a good score.
\n\n[/quote]
", "rawMarkdown": "Cool, now everyone's using Neon for this. Such an effective way to gain attention in a crowded space of deep learning tools... \r\n\r\nbut I think it's well worth it. Thanks for the generous sharing! Will be the 1st time to try Neon, but it seems cool.\r\n\r\n[quote=Anil Thomas;100902]\r\n\r\nI have been threatening to release the source code for a while. [Here][1] it is, finally! With 4 weeks left before the deadline, there's hopefully enough time for everyone to process the code and make improvements.\r\n\r\nAs given, it should reproduce my current score (which is at second place on the leaderboard). Note that you will see quite a bit of variation from run to run, so it could take multiple tries to get a good score.\r\n\r\n\r\n [1]: https://github.com/anlthms/whale-2015\r\n\r\n[/quote]\r\n" }, { "id": 100847, "postDate": "2015-12-11T03:22:47.713Z", "content": "[quote=Schurik;100633]
\n\nThanks @Anil Thomas for sharing your code. Could you tell me how to load my croped images into neon or which example is best for loading own imageset.
\n\n[/quote]
\n\nThe alexnet example is a good place to start. It requires that you pre-process the images with the provided batch writer utility.
", "rawMarkdown": "[quote=Schurik;100633]\r\n\r\nThanks @Anil Thomas for sharing your code. Could you tell me how to load my croped images into neon or which example is best for loading own imageset.\r\n\r\n[/quote]\r\n\r\nThe [alexnet example][1] is a good place to start. It requires that you pre-process the images with the provided [batch writer utility][2].\r\n\r\n\r\n [1]: https://github.com/NervanaSystems/neon/blob/master/examples/alexnet.py\r\n [2]: https://github.com/NervanaSystems/neon/blob/master/neon/util/batch_writer.py" }, { "id": 100633, "postDate": "2015-12-09T07:22:40.050Z", "content": "Thanks @Anil Thomas for sharing your code. Could you tell me how to load my croped images into neon or which example is best for loading own imageset.
", "rawMarkdown": "Thanks @Anil Thomas for sharing your code. Could you tell me how to load my croped images into neon or which example is best for loading own imageset." }, { "id": 100618, "postDate": "2015-12-09T00:55:49.857Z", "content": "Thanks @Anil Thomas for sharing this. Look forward to testing the code.
", "rawMarkdown": "Thanks @Anil Thomas for sharing this. Look forward to testing the code.\r\n" }, { "id": 100245, "postDate": "2015-12-04T06:02:57.347Z", "content": "[quote=rc;99530]
\n\nDo you train two separate networks to detect the two points or in a double-headed network? \n[/quote]
\n\nI trained two separate networks.
\n\n[quote]\nIs the mask a circle or a an ellipse centered at the point? How large should they be?\n[/quote]
\n\nI draw a circle (a few pixels wide) around the point of interest. Instead of setting those pixels to white, I set the intensities correlated to the proximity to the point of interest.
\n\nI will post a video recording of the entire meetup. That should make this somewhat clearer.
", "rawMarkdown": "[quote=rc;99530]\r\n\r\nDo you train two separate networks to detect the two points or in a double-headed network? \r\n[/quote]\r\n\r\nI trained two separate networks.\r\n\r\n[quote]\r\nIs the mask a circle or a an ellipse centered at the point? How large should they be?\r\n[/quote]\r\n\r\nI draw a circle (a few pixels wide) around the point of interest. Instead of setting those pixels to white, I set the intensities correlated to the proximity to the point of interest.\r\n\r\nI will post a video recording of the entire meetup. That should make this somewhat clearer.\r\n" }, { "id": 100244, "postDate": "2015-12-04T05:53:49.257Z", "content": "[quote=dietCoke;99452]
\n\nCan you give a sense for how accurate your autoencoder is?
\n\n[/quote]
\n\nThe crops are of varying quality. Most of them look reasonable. But there are many that are completely off. Here's a sample (also in the slide deck).\n
\n\nThis was good enough for a logloss of around 4.
", "rawMarkdown": "[quote=dietCoke;99452]\r\n\r\nCan you give a sense for how accurate your autoencoder is? \r\n\r\n[/quote]\r\n\r\nThe crops are of varying quality. Most of them look reasonable. But there are many that are completely off. Here's a sample (also in the slide deck).\r\n![cropped images][1]\r\n\r\nThis was good enough for a logloss of around 4.\r\n\r\n\r\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/100244/3355/crops.png?sv=2012-02-12&se=2015-12-07T05%3A53%3A56Z&sr=b&sp=r&sig=8vrWDCzl%2FigvULzzbfldmWNlFefnvOKMkg%2BGhNm1fbw%3D" }, { "id": 99530, "postDate": "2015-11-26T03:36:24.923Z", "content": "Thanks a lot for sharing!
\n\nDo you train two separate networks to detect the two points or in a double-headed network? Multi-task training usually performs better since multiple supervision signals are stronger than a single signal. If the latter is the case, which layer should the two heads branch?
\n\nThe slide presents two training methods with co-ordinates and with a mask. The SumSquared cost function used in the detection network indicates that the label actually used is the mask. How to generate the segmentation mask from the point coordinate? Is the mask a circle or a an ellipse centered at the point? How large should they be?
", "rawMarkdown": "Thanks a lot for sharing!\r\n\r\nDo you train two separate networks to detect the two points or in a double-headed network? Multi-task training usually performs better since multiple supervision signals are stronger than a single signal. If the latter is the case, which layer should the two heads branch?\r\n\r\nThe slide presents two training methods with co-ordinates and with a mask. The SumSquared cost function used in the detection network indicates that the label actually used is the mask. How to generate the segmentation mask from the point coordinate? Is the mask a circle or a an ellipse centered at the point? How large should they be?" }, { "id": 99452, "postDate": "2015-11-25T00:06:00.587Z", "content": "Can you give a sense for how accurate your autoencoder is?
\n\nAlso, thanks for the code!
", "rawMarkdown": "Can you give a sense for how accurate your autoencoder is? \r\n\r\nAlso, thanks for the code!" }, { "id": 99439, "postDate": "2015-11-24T21:10:24.590Z", "content": "Do we have permission to use the bonnet-tip and blowhead (point1 and point2) json files?
", "rawMarkdown": "Do we have permission to use the bonnet-tip and blowhead (point1 and point2) json files?" }, { "id": 101662, "postDate": "2015-12-16T10:15:03.257Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 100902, "author_name": "Anil Thomas", "author_url": "", "post_date": "2015-12-11T20:21:25.707000", "content": "I have been threatening to release the source code for a while. Here it is, finally! With 4 weeks left before the deadline, there's hopefully enough time for everyone to process the code and make improvements.
\n\nAs given, it should reproduce my current score (which is at second place on the leaderboard). Note that you will see quite a bit of variation from run to run, so it could take multiple tries to get a good score.
", "votes": 6, "replies": [] }, { "id": 100249, "author_name": "Anil Thomas", "author_url": "", "post_date": "2015-12-04T06:34:30.967000", "content": "Ok, here's the video from the meetup. I apologize in advance for my snail-paced delivery :-)
\n\nhttps://www.youtube.com/watch?v=WfuDrJA6JBE
\n\nWill post more of the code next week.
", "votes": 4, "replies": [] }, { "id": 103490, "author_name": "Shize Su", "author_url": "", "post_date": "2016-01-03T17:24:16.510000", "content": "[quote=Anil Thomas;100249]
\n\nOk, here's the video from the meetup. I apologize in advance for my snail-paced delivery :-)
\n\nhttps://www.youtube.com/watch?v=WfuDrJA6JBE
\n\nWill post more of the code next week.
\n\n[/quote]
\n\n@Anil: Hi Anil, it seems that your posted code to generate the test points is not reproducible. There is a seed parameter, but it doesn't seem to work...Would you be so kind to let us know how to make your posted code to generate the test points reproducible?
\n\nThanks in advance, and happy new year!
\n\nBest regards,
\n\nShize
", "votes": 1, "replies": [] }, { "id": 103235, "author_name": "lhatsk", "author_url": "", "post_date": "2015-12-30T14:15:10.507000", "content": "[quote=James King;102351]
\n\n"Heuristic estimate of test error" bounced around between 30 and 40 using imwidth = 224, no change to nchan or network topology. The localizer for point 2 stopped early, I forget which iteration. The test crops looked pretty good but still only got 5.51 on the leader board.
\n\n[/quote]
\n\nInteresting, I did the same thing, but my test error barely moves and when, gets worse:
\n\nEpoch 0 [Train |████████████████████| 142/142 batches, 1.58 cost, 98.77s]
\n\n
\nHeuristic estimate of test error 311.10Epoch 1 [Train |████████████████████| 142/142 batches, 1.51 cost, 98.74s]\nHeuristic estimate of test error 311.10
\n\nEpoch 2 [Train |████████████████████| 142/142 batches, 1.51 cost, 98.81s]\nHeuristic estimate of test error 311.10
\n\nEpoch 3 [Train |████████████████████| 142/142 batches, 1.51 cost, 98.10s]\nHeuristic estimate of test error 319.65
\n\nEpoch 4 [Train |████████████████████| 142/142 batches, 1.51 cost, 98.89s]
\n\nSame for the second point.
\n\n/edit: I may have found the culprit, didn't redo the preprocessing...
", "votes": 1, "replies": [] }, { "id": 102432, "author_name": "James King", "author_url": "", "post_date": "2015-12-22T16:48:30.743000", "content": "[quote=yilisg;102430]\nThanks James -- I kept the imwidth and had better luck playing with the nchan and network structure.\n[/quote]\nProbably a good idea. I can get good test crops with lower imwidth but the classifier doesn't run right. Cropping with small enough imwidth to fit in memory and then increasing imwidth back to 384 when classifying seems to work.
\n\nI ran some generic networks in cxxnet to try to classify the neon-produced crops but they did not perform well. I'm very surprised at how well Anil's classifier does with no preprocessing to remove all the irrelevant noise.
", "votes": 1, "replies": [] }, { "id": 102243, "author_name": "Kostas", "author_url": "", "post_date": "2015-12-20T23:08:47.490000", "content": "[quote=James King;102235]
\n\n[quote=James King;102233]
\n\nWell, I was able to produce some reasonable test crops with imwidth=192, but the classifier crashes when it passes P = -1 and Q = -1 to update_grid in kernel_specs.py.
\n\n[/quote]
\n\nI was able to fix the crash by changing
\n\nfor idx in range(6):\nto \nfor idx in range(5):but only got an lb score of 6.36. Maybe time to try a different approach.
\n\n[/quote]
\n\nFYI, I got a top 10 score based on modifications of Anil's code, using only a 4GB GPU.
", "votes": 1, "replies": [] }, { "id": 102163, "author_name": "Anil Thomas", "author_url": "", "post_date": "2015-12-20T03:50:10.467000", "content": "[quote]
\n\n@Anil is there a way to fit the localizer using less memory? It exhausts the memory on my 4GB gpu:
\n\n[/quote]
\n\nAs written, the localizer uses about 9GB of memory. You can reduce memory consumption by making the network narrower or shallower.
\n\nThere are two easy ways to make the network narrower:
\n\n1) Change the imwidth variable (this determines the dimensions of the input images) in
\n\nrun.shfrom 384 to something smaller. You should be able to get good results using 256. Even 192 might work. Note that you must redo the preparation step in this case. Check the README for instructions on how to do this.2) Change the nchan variable (this is the number of output channels in the conv and deconv layers) to something smaller. This number must be a multiple of 16.
\n\nMaking the network less deep is slightly trickier. There are two loops in localizer.py that adds conv and deconv layers. Currently they are hardcoded to execute 16 and 15 times respectively. You can try changing those numbers to something like 8 and 7, for example.
", "votes": 1, "replies": [] }, { "id": 102154, "author_name": "James King", "author_url": "", "post_date": "2015-12-20T02:27:03.953000", "content": "Was anyone able to produce test crops? With a batch size of 8 I get all the testpoints at 383, 383 (imgwidth - 1) so no crops can be produced - and I ran up to 40 epochs.
", "votes": 1, "replies": [] }, { "id": 101967, "author_name": "James King", "author_url": "", "post_date": "2015-12-18T05:07:28.040000", "content": "[quote=DataGeek;101965]\nIt would be helpful if you can tell how you solved all the problems you had during installation\n[/quote]
\n\nThere's not that much to say. After many futile efforts with my existing environment (couldn't find numpy, numpy was the wrong version, pycuda would not install correctly) I decided to create a fresh environment in a VM. This gave me a working install of neon, but then I realized the VM could not see the gpu. So I loaded Ubuntu trusty onto a usb and created a new partition with a completely clean install. Blacklisted nouveau, installed nvidia drivers, and installed cuda per NVIDIA's instructions
\n\nhttp://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/#axzz3ue67q5TI
\n\nThen I followed the instructions on github for installing neon (in a venv, not the sysinstall) and it worked on the first try.
", "votes": 1, "replies": [] }, { "id": 101963, "author_name": "", "author_url": "", "post_date": "2015-12-18T04:31:33.607000", "content": "I'm still unsure of what the input shapes are and if there are any augmentations being used in the neon version, but for anyone who's curious I'm able to get it to learn better (~3.4 on my holdout) if I use batch normalization and adadelta on a smaller network (1/4th the number of filters in each layer). Before I was using Nesterov SGD, so it seems batch norm and adadelta are necessary to some extent.
\n\n[quote=Jesse;101657]
\n\nCould someone explain what the input to the classifier looks like (shape, etc.)? The train crops ...
\n\n[/quote]
", "votes": 1, "replies": [] }, { "id": 101896, "author_name": "Kostas", "author_url": "", "post_date": "2015-12-17T18:12:37.800000", "content": "Thanks for sharing Anil. Great code and the Neon framework looks neat.\n I am experimenting with your encoder and I was wondering if in the LocalizerLoader there is a quicker way to set the mask other than passing floats one by one to the rather slow function call_compound_kernel() ?
", "votes": 1, "replies": [] }, { "id": 101698, "author_name": "Leon Yang", "author_url": "", "post_date": "2015-12-16T15:05:38.560000", "content": "@ Nina Chen Thank you for the solution. Actually I did make this change except I made a mistake by changing it into gcc...
\n\nAnyway thank you for telling me :D
\n\nP.S. the flag -Wno-sign-compare is also needed under line 47 of Makefile to avoid the signed/unsigned problem.
", "votes": 1, "replies": [] }, { "id": 101657, "author_name": "", "author_url": "", "post_date": "2015-12-16T10:01:17.973000", "content": "Could someone explain what the input to the classifier looks like (shape, etc.)? The train crops are of varying shapes (I was able to run the crop code to see what the output looked like), and from what I can tell the images are not being resized or reshaped before being fed to the network. An explanation would help, and some example images would be nice as well. I tried to install neon to run it, but I'm running into issues with the install.
\n\nJust FYI, I'm attempting to re-create this in lasagne but I'm failing to get my classifier to learn — my validation score never goes below ~5.2. I'm assuming it's because my image crops are bad / not the same, as I've ruled out any issues with labeling, etc. My train loss will continue to decrease, but validation loss does not. Maybe overfitting is an issue? I also didn't see any image augmentations to prevent overfitting in the neon implementation, but maybe something other than dropout is being done that I'm unaware of?
\n\nThank you in advance!
", "votes": 1, "replies": [] }, { "id": 101491, "author_name": "Anil Thomas", "author_url": "", "post_date": "2015-12-15T15:56:48.130000", "content": "[quote=Anil Thomas;101115]
\n\nHold on! Found a bug in the code. It turns out that the probability of getting a 3.5ish score in a single try is less than 0.25. Will push an update tomorrow that should make it more reliable.
\n\n[/quote]
\n\nA fix has been applied. Please update your copy of neon:
\n\n", "votes": 1, "replies": [] }, { "id": 101378, "author_name": "Anil Thomas", "author_url": "", "post_date": "2015-12-15T05:41:43.860000", "content": "cd neon\ngit pull\n[quote]
\n\nAny plans of supporting older GPUs/GPUs with compute capabilities < 5.0 on Neon?
\n\nMost of us can only get hold of grid K520 on EC2, for example.
\n\n[/quote]
\n\nSudeep, we are working on adding Kepler support back in. However, it is unlikely to be done in time for this competition.
", "votes": 1, "replies": [] }, { "id": 101365, "author_name": "Abhishek Thakur", "author_url": "", "post_date": "2015-12-15T04:07:48.027000", "content": "Cool.
\n\nDid you also get this error:
\n\n\n", "votes": 1, "replies": [] }, { "id": 101363, "author_name": "Abhishek Thakur", "author_url": "", "post_date": "2015-12-15T03:59:45.557000", "content": "2015-12-15 04:04:03,589 - neon.data.imageloader - ERROR - Unable to\n load loader.so. Ensure that this file has been compiled Traceback\n (most recent call last): File "./localizer.py", line 47, in \n point_num=point_num) File "/home/ubuntu/whale/whale-2015/localizer_loader.py", line 31, in\n init\n subset_pct, nlabels, macro, dtype) File "/usr/local/lib/python2.7/dist-packages/neon-1.1.3-py2.7.egg/neon/data/imageloader.py",\n line 73, in init\n self.start() File "/usr/local/lib/python2.7/dist-packages/neon-1.1.3-py2.7.egg/neon/data/imageloader.py",\n line 192, in start\n self.loader = self.loaderlib.start(ct.c_int(self.img_size), AttributeError: 'LocalizerLoader' object has no attribute 'loaderlib'
\n[quote=Sudeep Juvekar;100988]
\n\nAny plans of supporting older GPUs/GPUs with compute capabilities < 5.0 on Neon?
\n\nMost of us can only get hold of grid K520 on EC2, for example.
\n\n[/quote]
\n\nI have the same question
", "votes": 1, "replies": [] }, { "id": 101115, "author_name": "Anil Thomas", "author_url": "", "post_date": "2015-12-14T02:51:19.050000", "content": "Hold on! Found a bug in the code. It turns out that the probability of getting a 3.5ish score in a single try is less than 0.25. Will push an update tomorrow that should make it more reliable.
", "votes": 1, "replies": [] }, { "id": 100988, "author_name": "Sudeep Juvekar", "author_url": "", "post_date": "2015-12-12T17:24:13.313000", "content": "Any plans of supporting older GPUs/GPUs with compute capabilities < 5.0 on Neon?
\n\nMost of us can only get hold of grid K520 on EC2, for example.
", "votes": 1, "replies": [] }, { "id": 100262, "author_name": "Julian de Wit", "author_url": "", "post_date": "2015-12-04T09:15:25.760000", "content": "Hello Anil,\nOne month ago I took roughly the same approach. I still have pain on my wrist from all the clicking :)
\n\n
\nAlthough I did not do an encoder ( I regressed) my mugshots look roughly the same.
\nHowever, I abondoned the "project" since my holdout loss was roughly 5.4 so I thought that I was waayyy off with my approach.My classifier of the mughots to get the final predictions was just not good.
\n\n
\nNow looking at your presentation it seems like ths final classifier is just a formality..
\nFew layers, bit of dropout et voila.. However.. For me it seems like that part is the big kicker..Could you assess if my mugshots are worse then yours ? That might be the reason the my classifier did not work. Or might my neural network classifier not be up to par with your Neon classifier. (Which looks very interesting).
\n\nEdit.. There was a bug in my labeling code .. Excuse me
\n\nMugs of my holdout are attached.
\n\n", "votes": 1, "replies": [] }, { "id": 99444, "author_name": "Anil Thomas", "author_url": "", "post_date": "2015-11-24T21:57:16.473000", "content": "
[quote=Sudeep Juvekar;99439]
\n\nDo we have permission to use the bonnet-tip and blowhead (point1 and point2) json files?
\n\n[/quote]
\n\nAbsolutely! I hope they are useful.
", "votes": 1, "replies": [] }, { "id": 102346, "author_name": "yilisg", "author_url": "", "post_date": "2015-12-21T19:27:38.627000", "content": "Fantastic stuff Anil +1, this is the coolest stuff I've seen in a while :-)
\n\nFor those who ran the code, I am curious about your "heuristic estimate of test error" after 40 epochs (and how this changes with different imwidth, nchan or network depth if you've experimented). I was only able to run with imwidth=384 and nchan=16 due to memory limits (6G) and currently hovering around a test error of 61 (before it goes back higher). As a reference point, each epochs took 140s to run (so ~1.5hr for 40 epochs) and I'd like to experiment in the "right direction" (more epochs vs. smaller crops vs. shallower/deeper network layers).
\n\nPerhaps I should also pre-process the crops further... hmm... anyway not much of a whale enthusiastic myself but I am excited to learn and try the Neon RNN on another Kaggle competition!
\n\nLastly, it was rather painful to get the code running (eventually it worked for me on a fresh ubuntu 14.04 install -- PM me if anyone wants a step-by-step instruction I've put together).
", "votes": 2, "replies": [] }, { "id": 102014, "author_name": "Sudeep Juvekar", "author_url": "", "post_date": "2015-12-18T16:43:40.197000", "content": "Batch size is usually adjusted to reduce the GPU memory allocation. Try changing 'z32' to 'z8' (or even smaller like z4) in run.sh. Neon, though, does not allow low batch sizes and throws an assertionFailure. A hackish work-around is to find out which line causes assertion failure and commenting it out (#). It's still not guaranteed to work and might change results, but still worth a try.
", "votes": 2, "replies": [] }, { "id": 101717, "author_name": "Sudeep Juvekar", "author_url": "", "post_date": "2015-12-16T17:31:33.160000", "content": "If you are having difficulty with Neon because of older GPU/installation problems, I have translated Anil's models to Caffe. Initial commits are here: https://github.com/sjuvekar/Kaggle-Whales-Caffe.
\n\nThe code is a work-in-progress. I will soon update with an independent post about it.
", "votes": 2, "replies": [] }, { "id": 101364, "author_name": "Sudeep Juvekar", "author_url": "", "post_date": "2015-12-15T04:06:14.247000", "content": "In fact check this github issue: https://github.com/NervanaSystems/neon/issues/80
\n\nThey are probably working on it as we speak.
\n\nGreat Framework! Let's do our best to bump it over Caffe and Theano (and Torch!) here :)
", "votes": 2, "replies": [] }, { "id": 100448, "author_name": "Anil Thomas", "author_url": "", "post_date": "2015-12-07T16:04:40.717000", "content": "Hi Julian, your mugshots look great. Certainly worth a better score than 5.4. I was going to say there's probably a bug in your classifier, but it looks like you found it already. Good luck!
", "votes": 2, "replies": [] }, { "id": 102208, "author_name": "haiyuansun", "author_url": "", "post_date": "2015-12-20T12:55:03.037000", "content": "Traceback (most recent call last):\n File "crop.py", line 98, in \n pool.map(cropfunc, range(pcount))\n File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map\n return self.map_async(func, iterable, chunksize).get()\n File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get\n raise self._value\nTypeError: rotate() got an unexpected keyword argument 'center'\nwhat's the problem with?
", "votes": 0, "replies": [] }, { "id": 101965, "author_name": "DataGeek", "author_url": "", "post_date": "2015-12-18T04:52:03.793000", "content": "[quote=James King;101964]
\n\n[quote=Anil Thomas;101795]
\n\n[quote]
\n\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon...
\n\n[/quote]
\n\nJames, is this python3? Neon currently needs 2.7.
\n\nThere's some anaconda specific instructions in the docs that may be helpful.
\n\n[/quote]
\n\nI was finally able to get neon to build on a fresh install of Trusty (and confirmed it is using the gpu).
\n\n[/quote]\nIt would be helpful if you can tell how you solved all the problems you had during installation
", "votes": 0, "replies": [] }, { "id": 100267, "author_name": "rc", "author_url": "", "post_date": "2015-12-04T10:16:56.480000", "content": "Some codes from the previous human facial key points detection competition can be used on the two points annotations.
\n\n\n\nhttps://github.com/olddocks/caffe-facialkp
", "votes": 0, "replies": [] }, { "id": 101513, "author_name": "", "author_url": "", "post_date": "2015-12-15T17:36:56.503000", "content": "", "votes": 1, "replies": [] }, { "id": 102436, "author_name": "", "author_url": "", "post_date": "2015-12-22T17:04:22.567000", "content": "", "votes": -1, "replies": [] }, { "id": 102427, "author_name": "", "author_url": "", "post_date": "2015-12-22T15:50:25.003000", "content": "", "votes": -2, "replies": [] }, { "id": 119518, "author_name": "", "author_url": "", "post_date": "2016-05-11T05:26:27.513000", "content": "", "votes": 0, "replies": [] }, { "id": 103811, "author_name": "", "author_url": "", "post_date": "2016-01-06T18:18:10.797000", "content": "", "votes": 0, "replies": [] }, { "id": 103728, "author_name": "", "author_url": "", "post_date": "2016-01-05T21:13:31.237000", "content": "", "votes": 0, "replies": [] }, { "id": 103727, "author_name": "", "author_url": "", "post_date": "2016-01-05T21:07:26.893000", "content": "", "votes": 0, "replies": [] }, { "id": 103723, "author_name": "", "author_url": "", "post_date": "2016-01-05T20:47:25.660000", "content": "", "votes": 0, "replies": [] }, { "id": 103590, "author_name": "", "author_url": "", "post_date": "2016-01-04T15:56:05.867000", "content": "", "votes": 0, "replies": [] }, { "id": 103482, "author_name": "", "author_url": "", "post_date": "2016-01-03T16:06:28.933000", "content": "", "votes": 0, "replies": [] }, { "id": 103449, "author_name": "", "author_url": "", "post_date": "2016-01-03T00:06:31.093000", "content": "", "votes": 0, "replies": [] }, { "id": 103442, "author_name": "", "author_url": "", "post_date": "2016-01-02T23:02:22.540000", "content": "", "votes": 0, "replies": [] }, { "id": 103245, "author_name": "", "author_url": "", "post_date": "2015-12-30T16:58:59.123000", "content": "", "votes": 0, "replies": [] }, { "id": 103217, "author_name": "", "author_url": "", "post_date": "2015-12-30T09:52:11.100000", "content": "", "votes": 0, "replies": [] }, { "id": 103216, "author_name": "", "author_url": "", "post_date": "2015-12-30T09:41:37.057000", "content": "", "votes": 0, "replies": [] }, { "id": 103198, "author_name": "", "author_url": "", "post_date": "2015-12-30T03:21:44.637000", "content": "", "votes": 0, "replies": [] }, { "id": 102644, "author_name": "", "author_url": "", "post_date": "2015-12-24T05:18:39.973000", "content": "", "votes": 0, "replies": [] }, { "id": 102637, "author_name": "", "author_url": "", "post_date": "2015-12-24T03:05:00.740000", "content": "", "votes": 0, "replies": [] }, { "id": 102458, "author_name": "", "author_url": "", "post_date": "2015-12-22T19:37:29.380000", "content": "", "votes": 0, "replies": [] }, { "id": 102454, "author_name": "", "author_url": "", "post_date": "2015-12-22T18:53:22.823000", "content": "", "votes": 0, "replies": [] }, { "id": 102430, "author_name": "", "author_url": "", "post_date": "2015-12-22T16:33:50.863000", "content": "", "votes": 0, "replies": [] }, { "id": 102351, "author_name": "", "author_url": "", "post_date": "2015-12-21T20:32:02.193000", "content": "", "votes": 0, "replies": [] }, { "id": 102235, "author_name": "", "author_url": "", "post_date": "2015-12-20T20:02:12.887000", "content": "", "votes": 0, "replies": [] }, { "id": 102233, "author_name": "", "author_url": "", "post_date": "2015-12-20T19:32:11.683000", "content": "", "votes": 0, "replies": [] }, { "id": 102182, "author_name": "", "author_url": "", "post_date": "2015-12-20T08:35:46.850000", "content": "", "votes": 0, "replies": [] }, { "id": 102165, "author_name": "", "author_url": "", "post_date": "2015-12-20T03:57:30.367000", "content": "", "votes": 0, "replies": [] }, { "id": 102164, "author_name": "", "author_url": "", "post_date": "2015-12-20T03:55:36.830000", "content": "", "votes": 0, "replies": [] }, { "id": 102162, "author_name": "", "author_url": "", "post_date": "2015-12-20T03:25:14.977000", "content": "", "votes": 0, "replies": [] }, { "id": 102129, "author_name": "", "author_url": "", "post_date": "2015-12-19T18:26:23.070000", "content": "", "votes": 0, "replies": [] }, { "id": 102122, "author_name": "", "author_url": "", "post_date": "2015-12-19T16:37:47.723000", "content": "", "votes": 0, "replies": [] }, { "id": 102118, "author_name": "", "author_url": "", "post_date": "2015-12-19T16:26:32.593000", "content": "", "votes": 0, "replies": [] }, { "id": 102111, "author_name": "", "author_url": "", "post_date": "2015-12-19T14:59:45.237000", "content": "", "votes": 0, "replies": [] }, { "id": 102075, "author_name": "", "author_url": "", "post_date": "2015-12-19T04:27:56.223000", "content": "", "votes": 0, "replies": [] }, { "id": 102021, "author_name": "", "author_url": "", "post_date": "2015-12-18T17:40:18.663000", "content": "", "votes": 0, "replies": [] }, { "id": 102009, "author_name": "", "author_url": "", "post_date": "2015-12-18T15:10:18.527000", "content": "", "votes": 0, "replies": [] }, { "id": 101968, "author_name": "", "author_url": "", "post_date": "2015-12-18T05:25:15.527000", "content": "", "votes": 0, "replies": [] }, { "id": 101966, "author_name": "", "author_url": "", "post_date": "2015-12-18T04:54:42.397000", "content": "", "votes": 0, "replies": [] }, { "id": 101964, "author_name": "", "author_url": "", "post_date": "2015-12-18T04:42:49.750000", "content": "", "votes": 0, "replies": [] }, { "id": 101936, "author_name": "", "author_url": "", "post_date": "2015-12-17T23:44:16.920000", "content": "", "votes": 0, "replies": [] }, { "id": 101795, "author_name": "", "author_url": "", "post_date": "2015-12-17T04:30:42.417000", "content": "", "votes": 0, "replies": [] }, { "id": 101788, "author_name": "", "author_url": "", "post_date": "2015-12-17T03:30:06.917000", "content": "", "votes": 0, "replies": [] }, { "id": 101690, "author_name": "", "author_url": "", "post_date": "2015-12-16T14:14:32.997000", "content": "", "votes": 0, "replies": [] }, { "id": 101652, "author_name": "", "author_url": "", "post_date": "2015-12-16T09:21:09.790000", "content": "", "votes": 0, "replies": [] }, { "id": 101385, "author_name": "", "author_url": "", "post_date": "2015-12-15T05:58:37.663000", "content": "", "votes": 0, "replies": [] }, { "id": 101384, "author_name": "", "author_url": "", "post_date": "2015-12-15T05:56:37.543000", "content": "", "votes": 0, "replies": [] }, { "id": 101381, "author_name": "", "author_url": "", "post_date": "2015-12-15T05:47:52.960000", "content": "", "votes": 0, "replies": [] }, { "id": 101375, "author_name": "", "author_url": "", "post_date": "2015-12-15T04:56:18.923000", "content": "", "votes": 0, "replies": [] }, { "id": 101368, "author_name": "", "author_url": "", "post_date": "2015-12-15T04:21:13.657000", "content": "", "votes": 0, "replies": [] }, { "id": 101347, "author_name": "", "author_url": "", "post_date": "2015-12-15T02:56:33.100000", "content": "", "votes": 0, "replies": [] }, { "id": 101344, "author_name": "", "author_url": "", "post_date": "2015-12-15T02:32:18.027000", "content": "", "votes": 0, "replies": [] }, { "id": 101204, "author_name": "", "author_url": "", "post_date": "2015-12-14T18:58:45.717000", "content": "", "votes": 0, "replies": [] }, { "id": 101165, "author_name": "", "author_url": "", "post_date": "2015-12-14T13:58:57.473000", "content": "", "votes": 0, "replies": [] }, { "id": 101140, "author_name": "", "author_url": "", "post_date": "2015-12-14T09:19:20.243000", "content": "", "votes": 0, "replies": [] }, { "id": 100913, "author_name": "", "author_url": "", "post_date": "2015-12-11T23:41:02.193000", "content": "", "votes": 0, "replies": [] }, { "id": 100847, "author_name": "", "author_url": "", "post_date": "2015-12-11T03:22:47.713000", "content": "", "votes": 0, "replies": [] }, { "id": 100633, "author_name": "", "author_url": "", "post_date": "2015-12-09T07:22:40.050000", "content": "", "votes": 0, "replies": [] }, { "id": 100618, "author_name": "", "author_url": "", "post_date": "2015-12-09T00:55:49.857000", "content": "", "votes": 0, "replies": [] }, { "id": 100245, "author_name": "", "author_url": "", "post_date": "2015-12-04T06:02:57.347000", "content": "", "votes": 0, "replies": [] }, { "id": 100244, "author_name": "", "author_url": "", "post_date": "2015-12-04T05:53:49.257000", "content": "", "votes": 0, "replies": [] }, { "id": 99530, "author_name": "", "author_url": "", "post_date": "2015-11-26T03:36:24.923000", "content": "", "votes": 0, "replies": [] }, { "id": 99452, "author_name": "", "author_url": "", "post_date": "2015-11-25T00:06:00.587000", "content": "", "votes": 0, "replies": [] }, { "id": 99439, "author_name": "", "author_url": "", "post_date": "2015-11-24T21:10:24.590000", "content": "", "votes": 0, "replies": [] }, { "id": 101662, "author_name": "", "author_url": "", "post_date": "2015-12-16T10:15:03.257000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "99438": "I shared my method in a meetup last week. In order to keep the playing field level (and to comply with the rules!), the [slide deck][1] has been made available.\r\n\r\nWill post more information next week.\r\n\r\n [1]: https://github.com/anlthms/whale-2015/blob/master/object-recognition.pdf", "100902": "I have been threatening to release the source code for a while. [Here][1] it is, finally! With 4 weeks left before the deadline, there's hopefully enough time for everyone to process the code and make improvements.\r\n\r\nAs given, it should reproduce my current score (which is at second place on the leaderboard). Note that you will see quite a bit of variation from run to run, so it could take multiple tries to get a good score.\r\n\r\n\r\n [1]: https://github.com/anlthms/whale-2015", "100249": "Ok, here's the video from the meetup. I apologize in advance for my snail-paced delivery :-)\r\n\r\nhttps://www.youtube.com/watch?v=WfuDrJA6JBE\r\n\r\nWill post more of the code next week.", "103490": "[quote=Anil Thomas;100249]\r\n\r\nOk, here's the video from the meetup. I apologize in advance for my snail-paced delivery :-)\r\n\r\nhttps://www.youtube.com/watch?v=WfuDrJA6JBE\r\n\r\nWill post more of the code next week.\r\n\r\n[/quote]\r\n\r\n@Anil: Hi Anil, it seems that your posted code to generate the test points is not reproducible. There is a seed parameter, but it doesn't seem to work...Would you be so kind to let us know how to make your posted code to generate the test points reproducible?\r\n\r\nThanks in advance, and happy new year!\r\n\r\nBest regards,\r\n\r\nShize\r\n\r\n\r\n", "103235": "[quote=James King;102351]\r\n\r\n\"Heuristic estimate of test error\" bounced around between 30 and 40 using imwidth = 224, no change to nchan or network topology. The localizer for point 2 stopped early, I forget which iteration. The test crops looked pretty good but still only got 5.51 on the leader board.\r\n\r\n[/quote]\r\n\r\nInteresting, I did the same thing, but my test error barely moves and when, gets worse:\r\n\r\nEpoch 0 [Train |████████████████████| 142/142 batches, 1.58 cost, 98.77s] \r\nHeuristic estimate of test error 311.10\r\n\r\nEpoch 1 [Train |████████████████████| 142/142 batches, 1.51 cost, 98.74s]\r\nHeuristic estimate of test error 311.10\r\n\r\nEpoch 2 [Train |████████████████████| 142/142 batches, 1.51 cost, 98.81s]\r\nHeuristic estimate of test error 311.10\r\n\r\nEpoch 3 [Train |████████████████████| 142/142 batches, 1.51 cost, 98.10s]\r\nHeuristic estimate of test error 319.65\r\n\r\nEpoch 4 [Train |████████████████████| 142/142 batches, 1.51 cost, 98.89s]\r\n\r\n\r\nSame for the second point.\r\n\r\n\r\n/edit: I may have found the culprit, didn't redo the preprocessing...", "102432": "[quote=yilisg;102430]\r\nThanks James -- I kept the imwidth and had better luck playing with the nchan and network structure.\r\n[/quote]\r\nProbably a good idea. I can get good test crops with lower imwidth but the classifier doesn't run right. Cropping with small enough imwidth to fit in memory and then increasing imwidth back to 384 when classifying seems to work.\r\n\r\nI ran some generic networks in cxxnet to try to classify the neon-produced crops but they did not perform well. I'm very surprised at how well Anil's classifier does with no preprocessing to remove all the irrelevant noise.", "102243": "[quote=James King;102235]\r\n\r\n[quote=James King;102233]\r\n\r\nWell, I was able to produce some reasonable test crops with imwidth=192, but the classifier crashes when it passes P = -1 and Q = -1 to update_grid in kernel_specs.py.\r\n\r\n[/quote]\r\n\r\nI was able to fix the crash by changing `for idx in range(6):` \r\nto \r\n`for idx in range(5):`\r\n\r\nbut only got an lb score of 6.36. Maybe time to try a different approach.\r\n\r\n[/quote]\r\n\r\nFYI, I got a top 10 score based on modifications of Anil's code, using only a 4GB GPU.", "102163": "[quote]\r\n\r\n@Anil is there a way to fit the localizer using less memory? It exhausts the memory on my 4GB gpu:\r\n\r\n[/quote]\r\n\r\nAs written, the localizer uses about 9GB of memory. You can reduce memory consumption by making the network narrower or shallower.\r\n\r\nThere are two easy ways to make the network narrower:\r\n\r\n1) Change the imwidth variable (this determines the dimensions of the input images) in `run.sh` from 384 to something smaller. You should be able to get good results using 256. Even 192 might work. Note that you must redo the preparation step in this case. Check the [README][1] for instructions on how to do this.\r\n\r\n2) Change the nchan variable (this is the number of output channels in the conv and deconv layers) to something smaller. This number must be a multiple of 16.\r\n\r\nMaking the network less deep is slightly trickier. There are two loops in [localizer.py][2] that adds conv and deconv layers. Currently they are hardcoded to execute 16 and 15 times respectively. You can try changing those numbers to something like 8 and 7, for example.\r\n\r\n\r\n [1]: https://github.com/anlthms/whale-2015/blob/master/README.md\r\n [2]: https://github.com/anlthms/whale-2015/blob/master/localizer.py", "102154": "Was anyone able to produce test crops? With a batch size of 8 I get all the testpoints at 383, 383 (imgwidth - 1) so no crops can be produced - and I ran up to 40 epochs.", "101967": "[quote=DataGeek;101965]\r\nIt would be helpful if you can tell how you solved all the problems you had during installation\r\n[/quote]\r\n\r\nThere's not that much to say. After many futile efforts with my existing environment (couldn't find numpy, numpy was the wrong version, pycuda would not install correctly) I decided to create a fresh environment in a VM. This gave me a working install of neon, but then I realized the VM could not see the gpu. So I loaded Ubuntu trusty onto a usb and created a new partition with a completely clean install. Blacklisted nouveau, installed nvidia drivers, and installed cuda per NVIDIA's instructions \r\n\r\nhttp://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/#axzz3ue67q5TI\r\n\r\nThen I followed the instructions on github for installing neon (in a venv, not the sysinstall) and it worked on the first try.\r\n", "101963": "I'm still unsure of what the input shapes are and if there are any augmentations being used in the neon version, but for anyone who's curious I'm able to get it to learn better (~3.4 on my holdout) if I use batch normalization and adadelta on a smaller network (1/4th the number of filters in each layer). Before I was using Nesterov SGD, so it seems batch norm and adadelta are necessary to some extent.\r\n\r\n[quote=Jesse;101657]\r\n\r\nCould someone explain what the input to the classifier looks like (shape, etc.)? The train crops ...\r\n\r\n[/quote]\r\n", "101896": "Thanks for sharing Anil. Great code and the Neon framework looks neat.\r\n I am experimenting with your encoder and I was wondering if in the LocalizerLoader there is a quicker way to set the mask other than passing floats one by one to the rather slow function call_compound_kernel() ?\r\n", "101698": "@ Nina Chen Thank you for the solution. Actually I did make this change except I made a mistake by changing it into gcc...\r\n\r\nAnyway thank you for telling me :D\r\n\r\nP.S. the flag -Wno-sign-compare is also needed under line 47 of Makefile to avoid the signed/unsigned problem.", "101657": "Could someone explain what the input to the classifier looks like (shape, etc.)? The train crops are of varying shapes (I was able to run the crop code to see what the output looked like), and from what I can tell the images are not being resized or reshaped before being fed to the network. An explanation would help, and some example images would be nice as well. I tried to install neon to run it, but I'm running into issues with the install.\r\n\r\nJust FYI, I'm attempting to re-create this in lasagne but I'm failing to get my classifier to learn — my validation score never goes below ~5.2. I'm assuming it's because my image crops are bad / not the same, as I've ruled out any issues with labeling, etc. My train loss will continue to decrease, but validation loss does not. Maybe overfitting is an issue? I also didn't see any image augmentations to prevent overfitting in the neon implementation, but maybe something other than dropout is being done that I'm unaware of?\r\n\r\nThank you in advance!", "101491": "[quote=Anil Thomas;101115]\r\n\r\nHold on! Found a bug in the code. It turns out that the probability of getting a 3.5ish score in a single try is less than 0.25. Will push an update tomorrow that should make it more reliable.\r\n\r\n[/quote]\r\n\r\nA fix has been applied. Please update your copy of neon:\r\n\r\n cd neon\r\n git pull\r\n\r\n", "101378": "[quote]\r\n\r\nAny plans of supporting older GPUs/GPUs with compute capabilities < 5.0 on Neon? \r\n\r\nMost of us can only get hold of grid K520 on EC2, for example. \r\n\r\n[/quote]\r\n\r\nSudeep, we are working on adding Kepler support back in. However, it is unlikely to be done in time for this competition.", "101365": "Cool.\r\n\r\nDid you also get this error:\r\n\r\n> 2015-12-15 04:04:03,589 - neon.data.imageloader - ERROR - Unable to\r\n> load loader.so. Ensure that this file has been compiled Traceback\r\n> (most recent call last): File \"./localizer.py\", line 47, in\r\n> point_num=point_num) File \"/home/ubuntu/whale/whale-2015/localizer_loader.py\", line 31, in\r\n> __init__\r\n> subset_pct, nlabels, macro, dtype) File \"/usr/local/lib/python2.7/dist-packages/neon-1.1.3-py2.7.egg/neon/data/imageloader.py\",\r\n> line 73, in __init__\r\n> self.start() File \"/usr/local/lib/python2.7/dist-packages/neon-1.1.3-py2.7.egg/neon/data/imageloader.py\",\r\n> line 192, in start\r\n> self.loader = self.loaderlib.start(ct.c_int(self.img_size), AttributeError: 'LocalizerLoader' object has no attribute 'loaderlib'\r\n\r\n", "101363": "[quote=Sudeep Juvekar;100988]\r\n\r\nAny plans of supporting older GPUs/GPUs with compute capabilities < 5.0 on Neon? \r\n\r\nMost of us can only get hold of grid K520 on EC2, for example. \r\n\r\n[/quote]\r\n\r\nI have the same question", "101115": "Hold on! Found a bug in the code. It turns out that the probability of getting a 3.5ish score in a single try is less than 0.25. Will push an update tomorrow that should make it more reliable.", "100988": "Any plans of supporting older GPUs/GPUs with compute capabilities < 5.0 on Neon? \r\n\r\nMost of us can only get hold of grid K520 on EC2, for example. ", "100262": "Hello Anil,\r\nOne month ago I took roughly the same approach. I still have pain on my wrist from all the clicking :)
\r\nAlthough I did not do an encoder ( I regressed) my mugshots look roughly the same.
\r\nHowever, I abondoned the \"project\" since my holdout loss was roughly 5.4 so I thought that I was waayyy off with my approach.
\r\n\r\nMy classifier of the mughots to get the final predictions was just not good.
\r\nNow looking at your presentation it seems like ths final classifier is just a formality..
\r\nFew layers, bit of dropout et voila.. However.. For me it seems like that part is the big kicker..
\r\n\r\nCould you assess if my mugshots are worse then yours ? That might be the reason the my classifier did not work. Or might my neural network classifier not be up to par with your Neon classifier. (Which looks very interesting).
\r\n\r\n**Edit.. There was a bug in my labeling code .. Excuse me**\r\n\r\nMugs of my holdout are attached.\r\n\r\n![enter image description here][1]\r\n\r\n\r\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/100262/3356/mugs.png?sv=2012-02-12&se=2015-12-08T10%3A04%3A50Z&sr=b&sp=r&sig=fqNTJBwP4VLU42xzwpBZGI4XrlrGy0B3bdOAsf%2BreJI%3D", "99444": "[quote=Sudeep Juvekar;99439]\r\n\r\nDo we have permission to use the bonnet-tip and blowhead (point1 and point2) json files?\r\n\r\n[/quote]\r\n\r\nAbsolutely! I hope they are useful.", "102346": "Fantastic stuff Anil +1, this is the coolest stuff I've seen in a while :-)\r\n\r\nFor those who ran the code, I am curious about your \"heuristic estimate of test error\" after 40 epochs (and how this changes with different imwidth, nchan or network depth if you've experimented). I was only able to run with imwidth=384 and nchan=16 due to memory limits (6G) and currently hovering around a test error of 61 (before it goes back higher). As a reference point, each epochs took 140s to run (so ~1.5hr for 40 epochs) and I'd like to experiment in the \"right direction\" (more epochs vs. smaller crops vs. shallower/deeper network layers).\r\n\r\nPerhaps I should also pre-process the crops further... hmm... anyway not much of a whale enthusiastic myself but I am excited to learn and try the Neon RNN on another Kaggle competition!\r\n\r\nLastly, it was rather painful to get the code running (eventually it worked for me on a fresh ubuntu 14.04 install -- PM me if anyone wants a step-by-step instruction I've put together).", "102014": "Batch size is usually adjusted to reduce the GPU memory allocation. Try changing 'z32' to 'z8' (or even smaller like z4) in run.sh. Neon, though, does not allow low batch sizes and throws an assertionFailure. A hackish work-around is to find out which line causes assertion failure and commenting it out (#). It's still not guaranteed to work and might change results, but still worth a try.", "101717": "If you are having difficulty with Neon because of older GPU/installation problems, I have translated Anil's models to Caffe. Initial commits are here: https://github.com/sjuvekar/Kaggle-Whales-Caffe.\r\n\r\nThe code is a work-in-progress. I will soon update with an independent post about it.", "101364": "In fact check this github issue: https://github.com/NervanaSystems/neon/issues/80\r\n\r\nThey are probably working on it as we speak. \r\n\r\nGreat Framework! Let's do our best to bump it over Caffe and Theano (and Torch!) here :)", "100448": "Hi Julian, your mugshots look great. Certainly worth a better score than 5.4. I was going to say there's probably a bug in your classifier, but it looks like you found it already. Good luck!", "102208": "Traceback (most recent call last):\r\n File \"crop.py\", line 98, in\r\n pool.map(cropfunc, range(pcount))\r\n File \"/usr/lib/python2.7/multiprocessing/pool.py\", line 251, in map\r\n return self.map_async(func, iterable, chunksize).get()\r\n File \"/usr/lib/python2.7/multiprocessing/pool.py\", line 558, in get\r\n raise self._value\r\nTypeError: rotate() got an unexpected keyword argument 'center'\r\nwhat's the problem with?", "101965": "[quote=James King;101964]\r\n\r\n[quote=Anil Thomas;101795]\r\n\r\n[quote]\r\n\r\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon...\r\n\r\n[/quote]\r\n\r\nJames, is this python3? Neon currently needs 2.7.\r\n\r\nThere's some [anaconda specific instructions][1] in the docs that may be helpful.\r\n\r\n\r\n [1]: http://neon.nervanasys.com/docs/latest/user_guide.html#anaconda\r\n\r\n[/quote]\r\n\r\nI was finally able to get neon to build on a fresh install of Trusty (and confirmed it is using the gpu).\r\n\r\n[/quote]\r\nIt would be helpful if you can tell how you solved all the problems you had during installation", "100267": "Some codes from the previous human facial key points detection competition can be used on the two points annotations. \r\n\r\nhttp://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/\r\n\r\nhttps://github.com/olddocks/caffe-facialkp", "101513": "Anil, thanks for sharing the code! \r\n\r\nI suppose we cannot run current neon release on AWS GPU instances. Can we have access to Nervana Cloud? Can you suggest other good options.", "102436": "[quote=James King;102432]\r\n\r\n[quote=yilisg;102430]\r\nThanks James -- I kept the imwidth and had better luck playing with the nchan and network structure.\r\n[/quote]\r\nProbably a good idea. I can get good test crops with lower imwidth but the classifier doesn't run right. Cropping with small enough imwidth to fit in memory and then increasing imwidth back to 384 when classifying seems to work.\r\n\r\nI ran some generic networks in cxxnet to try to classify the neon-produced crops but they did not perform well. I'm very surprised at how well Anil's classifier does with no preprocessing to remove all the irrelevant noise.\r\n\r\n[/quote]\r\ndelete.", "102427": "Can you push testpoint1.json and testpoint2 in github please?I don't have GPU and need nearly 4000 hours to running the code ,I can't cropped it done untill deadline.", "119518": "can anyone help i extract the head of whale but i dont know how to align the faces of whale please give suggestion how to align .\r\nthanks \r\nmouzmeen\r\n", "103811": "@anil I am getting this error:\r\n\r\nTraceback (most recent call last):\r\n File \"./classifier.py\", line 63, in \r\n callbacks=callbacks)\r\n File \"/home/ubuntu/wa/1/neon/models/model.py\", line 120, in fit\r\n self._epoch_fit(dataset, callbacks)\r\n File \"/home/ubuntu/wa/1/neon/models/model.py\", line 141, in _epoch_fit\r\n x = self.fprop(x)\r\n File \"/home/ubuntu/wa/1/neon/models/model.py\", line 171, in fprop\r\n return self.layers.fprop(x, inference)\r\n File \"/home/ubuntu/wa/1/neon/layers/container.py\", line 103, in fprop\r\n x = l.fprop(x, inference)\r\n File \"/home/ubuntu/wa/1/neon/layers/layer.py\", line 527, in fprop\r\n self.be.fprop_conv(self.nglayer, inputs, self.W, self.outputs, bsum=self.batch_sum)\r\n File \"/home/ubuntu/wa/whale-2015/neon/backends/nervanacpu.py\", line 960, in fprop_conv\r\n array_O = O.get().reshape(layer.dimO)\r\nValueError: can only specify one unknown dimension\r\n\r\nThese are the parameters at this point:\r\n\r\n**layer**: neon.backends.layer_cpu.ConvLayer object at 0x7f57eb242ad0\r\n\r\n**O:** CPUTensor(base 0x7f57eb2b4260) name:None shape:(1024, 32) dtype: type 'numpy.float32' strides:(128, 4) is_c_contiguous:True)\r\n\r\n**dim:** (1024, 1, -1, -1, 32))", "103728": "[quote=James King;103590]\r\n\r\nIs there any way to make neon print both train and test error with each iteration? The callback method only takes one evaluation set as an argument.\r\n\r\n[/quote]\r\n\r\nDoesn't the progress bar already print out the train error? If you are trying to add a different kind of error metric, maybe define a new Callback class and use the Callbacks.add_callback() function to hook it up.", "103727": "[quote=Shize Su;103490]\r\n\r\n@Anil: Hi Anil, it seems that your posted code to generate the test points is not reproducible. There is a seed parameter, but it doesn't seem to work...Would you be so kind to let us know how to make your posted code to generate the test points reproducible?\r\n\r\n[/quote]\r\n\r\nHi Shize, you're right about the results not being exactly reproducible. Your best bet at this point may be to ensemble multiple results to end up with something more stable.\r\n\r\nWe will have a deterministic GPU backend in neon within a couple of weeks.", "103723": "[quote=loweew;103449]\r\n\r\nHow straight forward is it to extract features using neon? I have a reasonable model trained, and would like to extract the penultimate layer outputs to use as inputs to a different ML method. \r\n\r\n[/quote]\r\n\r\nYou should be able to directly access the *outputs* member of the layer that you are interested in. See the function *get_outputs*() inside [evaluator.py][1] for an example of getting the output of the network. In this example, I get the output of the final layer. If you are interested in the penultimate layer, you might want to do something like: *model.layers.layers[-2].outputs.get().*\r\n\r\n\r\n [1]: https://github.com/anlthms/whale-2015/blob/master/evaluator.py", "103590": "Is there any way to make neon print both train and test error with each iteration? The callback method only takes one evaluation set as an argument.", "103482": "Now tried to change the imwidth=256 , but getting the below error in Classifying:\r\n\r\n File \"/home/roshan/anaconda3/envs/python2/lib/python2.7/site-packages/neon/backends/layer_gpu.py\", line 1122, in _magic64\r\n magic, shift = _magic32(nmax, d)\r\n File \"/home/roshan/anaconda3/envs/python2/lib/python2.7/site-packages/neon/backends/layer_gpu.py\", line 1103, in _magic32\r\n nc = ((nmax + 1) // d) * d - 1\r\nZeroDivisionError: integer division or modulo by zero\r\n", "103449": "How straight forward is it to extract features using neon? I have a reasonable model trained, and would like to extract the penultimate layer outputs to use as inputs to a different ML method. I couldn't find any working examples or tutorials for this. Any input would be greatly appreciated.", "103442": "lhatsk, if you reduce the imwidth parameter, make sure that the network doesn't squish down the spatial dimensions to invalid values. You can get the network structure printed out by passing ***-v*** to classifier.py (this only works for valid networks). Maybe you can unroll the for loop within [classifier.py][1] and delete a couple of the maxpooling layers to prevent the dimensions from getting too small as you forward prop the images. Another possibility is setting the strides parameter to 1 in the first layer.\r\n\r\n\r\n [1]: https://github.com/anlthms/whale-2015/blob/master/classifier.py", "103245": "OK, test error decreases, I get crops but the classifier aborts now. Has anyone an idea? Didn't change anything in the code besides imwidth. Thanks!\r\n\r\n/edit: Post #67 (James King) refers to the same problem and solves it. \r\n\r\nTraceback (most recent call last):\r\n\r\n File \"./classifier.py\", line 63, in \r\n callbacks=callbacks)\r\n\r\n File \"/usr/local/lib/python2.7/dist-packages/neon/models/model.py\", line 109, in fit\r\n self.initialize(dataset, cost)\r\n\r\n File \"/usr/local/lib/python2.7/dist-packages/neon/models/model.py\", line 73, in initialize\r\n prev_input = self.layers.configure(prev_input)\r\n\r\n File \"/usr/local/lib/python2.7/dist-packages/neon/layers/container.py\", line 70, in configure\r\n in_obj = l.configure(in_obj)\r\n\r\n File \"/usr/local/lib/python2.7/dist-packages/neon/layers/layer.py\", line 516, in configure\r\n self.nglayer = self.be.conv_layer(self.be.default_dtype, **self.convparams)\r\n\r\n File \"/usr/local/lib/python2.7/dist-packages/neon/backends/nervanagpu.py\", line 1448, in conv_layer\r\n relu, bsum, deterministic_update)\r\n\r\n File \"/usr/local/lib/python2.7/dist-packages/neon/backends/layer_gpu.py\", line 525, in __init__\r\n sm_count)\r\n\r\n File \"/usr/local/lib/python2.7/dist-packages/neon/backends/kernel_specs.py\", line 549, in update_grid\r\n return (grid[0][0], grid[0][1], threads)\r\n\r\nIndexError: list index out of range\r\n\r\n", "103217": "[quote=yilisg;102346]\r\n\r\nFantastic stuff Anil +1, this is the coolest stuff I've seen in a while :-)\r\n\r\nFor those who ran the code, I am curious about your \"heuristic estimate of test error\" after 40 epochs (and how this changes with different imwidth, nchan or network depth if you've experimented). I was only able to run with imwidth=384 and nchan=16 due to memory limits (6G) and currently hovering around a test error of 61 (before it goes back higher). As a reference point, each epochs took 140s to run (so ~1.5hr for 40 epochs) and I'd like to experiment in the \"right direction\" (more epochs vs. smaller crops vs. shallower/deeper network layers).\r\n\r\nPerhaps I should also pre-process the crops further... hmm... anyway not much of a whale enthusiastic myself but I am excited to learn and try the Neon RNN on another Kaggle competition!\r\n\r\nLastly, it was rather painful to get the code running (eventually it worked for me on a fresh ubuntu 14.04 install -- PM me if anyone wants a step-by-step instruction I've put together).\r\n\r\n[/quote]\r\nHi yilisg, i also installed a fresh ubuntu install in VM-Workstation. my code starts to run but fails around traincrops part. could you please send me your step-by-step instructions for the install ? \r\nmy email is mpsampat@gmail.com", "103216": "[quote=haiyuansun;102208]\r\n\r\nTraceback (most recent call last):\r\n File \"crop.py\", line 98, in \r\n pool.map(cropfunc, range(pcount))\r\n File \"/usr/lib/python2.7/multiprocessing/pool.py\", line 251, in map\r\n return self.map_async(func, iterable, chunksize).get()\r\n File \"/usr/lib/python2.7/multiprocessing/pool.py\", line 558, in get\r\n raise self._value\r\nTypeError: rotate() got an unexpected keyword argument 'center'\r\nwhat's the problem with?\r\n\r\n[/quote]\r\nHi haiyuansun, I have the exact same error. were you able to resolve this issue ? Is yes could you me how you resolved it ? ", "103198": "late to this, but just try to learn how to run deep learning on gpu. my computer has cuda 5.0, but when I try the script, it tells me device 0 does not have cuda compute capability 5.0 or greater. \r\nRuntimeError: Device 0 does not have CUDA compute capability 5.0 or greater\r\n\r\n![enter image description here][1]\r\n\r\nmy device config\r\n [1]: http://C:%5CUsers%5Cli%5CDesktop%5Cgraph.png\r\n", "102644": "[quote=MrTwiggy;102637]\r\n\r\nIs there support for non-Maxwell GPUs in neon? My main goal is to reproduce the training and test crops to be able to train my own model for identification on cropped images, but installing neon and getting it to work has proven to be insurmountable. I have a Geforce GTX 780 btw, which is Kepler afaik. Has anyone publicly posted a set of JSON points for train/test sets on the bonnet and blowhole for cropping?\r\n\r\n[/quote]\r\n\r\nSee: https://github.com/NervanaSystems/neon/issues/80\r\n\r\nI don't think non-Maxwell support will be ready for this competition.", "102637": "Is there support for non-Maxwell GPUs in neon? My main goal is to reproduce the training and test crops to be able to train my own model for identification on cropped images, but installing neon and getting it to work has proven to be insurmountable. I have a Geforce GTX 780 btw, which is Kepler afaik. Has anyone publicly posted a set of JSON points for train/test sets on the bonnet and blowhole for cropping?", "102458": "[quote=jwjohnson314;102454]\r\n\r\n[quote=Sudeep Juvekar;101717]\r\n\r\nIf you are having difficulty with Neon because of older GPU/installation problems, I have translated Anil's models to Caffe. Initial commits are here: https://github.com/sjuvekar/Kaggle-Whales-Caffe.\r\n\r\nThe code is a work-in-progress. I will soon update with an independent post about it.\r\n\r\n[/quote]\r\n\r\nSudeep, have you had success running the Caffe code? I run out of memory on a K40 with 12gb, even after removing layers from the model.\r\n\r\n[/quote]\r\n\r\nClassifier runs fine with cropped training images (classifier_solver.prototxt in latest commit in solver/ or last line in run.sh). Cropping code is not complete yet, some people have volunteered to modify it.", "102454": "[quote=Sudeep Juvekar;101717]\r\n\r\nIf you are having difficulty with Neon because of older GPU/installation problems, I have translated Anil's models to Caffe. Initial commits are here: https://github.com/sjuvekar/Kaggle-Whales-Caffe.\r\n\r\nThe code is a work-in-progress. I will soon update with an independent post about it.\r\n\r\n[/quote]\r\n\r\nSudeep, have you had success running the Caffe code? I run out of memory on a K40 with 12gb, even after removing layers from the model.", "102430": "Thanks James -- I kept the imwidth and had better luck playing with the nchan and network structure.\r\n\r\n@Anil Xmas came early and I've just added a second GPU to my machine. On other software, I was able to use the entire 12gb (2x6gb), but in neon I am still getting the error (cuMemAlloc failed: out of memory). I understand since v0.9 neon supports multiple GPUs -- but after I enable -bmgpu I get the following error (\"No module named mgpu.nervanamgpu\"). Kindly let me know what should I do to enable two GPUs (perhaps a special kaggler release where max_devices is capped at 2?). Thanks!\r\n\r\nIn the meantime, I suppose I could run two processes in parallel, each assigned to a different GPU but that wouldn't solve my ram limitation :(", "102351": "\"Heuristic estimate of test error\" bounced around between 30 and 40 using imwidth = 224, no change to nchan or network topology. The localizer for point 2 stopped early, I forget which iteration. The test crops looked pretty good but still only got 5.51 on the leader board.", "102235": "[quote=James King;102233]\r\n\r\nWell, I was able to produce some reasonable test crops with imwidth=192, but the classifier crashes when it passes P = -1 and Q = -1 to update_grid in kernel_specs.py.\r\n\r\n[/quote]\r\n\r\nI was able to fix the crash by changing `for idx in range(6):` \r\nto \r\n`for idx in range(5):`\r\n\r\nbut only got an lb score of 6.36. Maybe time to try a different approach.", "102233": "Well, I was able to produce some reasonable test crops with imwidth=192, but the classifier crashes when it passes P = -1 and Q = -1 to update_grid in kernel_specs.py.", "102182": "[quote=Anil Thomas;102163]\r\n\r\nMaking the network less deep is slightly trickier. There are two loops in [localizer.py][2] that adds conv and deconv layers. Currently they are hardcoded to execute 16 and 15 times respectively. You can try changing those numbers to something like 8 and 7, for example.\r\n\r\n [2]: https://github.com/anlthms/whale-2015/blob/master/localizer.py\r\n\r\n[/quote]\r\n\r\nThanks Anil, this helps. Considering that the localizer networks are not learning any high level/abstract representations, how important is depth in this case?\r\n\r\nIt will be interesting to find the shallowest network capable of learning bonnet/blowhole points, worth running a few experiments.", "102165": "[quote]\r\n\r\n@lhatsk, I too had the same issues with loader folder , and after that faced cuMemAlloc failed issue. Changed 'z32' to 'z8' as Sudeep suggested (with 4GB Gpu) and its running fine now, but slow...currently in 9th Epoch and it took around 20 hours :(\r\n\r\n[/quote]\r\n\r\n@RKR, that sounds like the code is running on the CPU. Try running `nvidia-smi` to see if the GPU is being utilized at all.", "102164": "[quote]\r\n\r\nBatch size is usually adjusted to reduce the GPU memory allocation. Try changing 'z32' to 'z8' (or even smaller like z4) in run.sh. Neon, though, does not allow low batch sizes and throws an assertionFailure. A hackish work-around is to find out which line causes assertion failure and commenting it out (#). It's still not guaranteed to work and might change results, but still worth a try.\r\n\r\n[/quote]\r\n\r\nI wouldn't recommend that. The assertion is present in the code for a good reason. If you are running this on a CPU, you can set the batch size to whatever you like, but on a GPU the options are limited.\r\n", "102162": "[quote]\r\n\r\nThanks for sharing Anil. Great code and the Neon framework looks neat.\r\n I am experimenting with your encoder and I was wondering if in the LocalizerLoader there is a quicker way to set the mask other than passing floats one by one to the rather slow function call_compound_kernel() ?\r\n\r\n[/quote]\r\n\r\nGreat to know you are having a good experience with neon.\r\n\r\nInstead of setting each pixel individually, you could make the target circle in advance. Keep that image in GPU memory and copy it to the appropriate location within the mask as needed.\r\n\r\nMost of the compute time is spent training the network, so you are unlikely to save more than a few minutes by optimizing the data loading part.", "102129": "its around 12.14 , but its taking 4900 seconds for each epoch :(", "102122": "@RKR Yes, changing it to 'z8' works. I just ran two epochs, each took around 900s. Any idea how much impact the batch size change has on the performance? The first errors were around 12.7...", "102118": "@lhatsk, I too had the same issues with loader folder , and after that faced cuMemAlloc failed issue. Changed 'z32' to 'z8' as Sudeep suggested (with 4GB Gpu) and its running fine now, but slow...currently in 9th Epoch and it took around 20 hours :(", "102111": "I had the same problem with loader.so, it looks like everything was build during make sysinstall, but the loader directory isn't copied to /usr/local/lib/python2.7/dist-packages/neon/data in my case. So I created a loader directory in data and copied over loader.so. I'm now at the next stage with cuMemAlloc failed: out of memory :-)", "102075": "The progress bar in this code is top notch :)", "102021": "I reduced `nchan` to the point where the out of memory error went away, but then got an nvcc compile error\r\n\r\n kernel.cu(26): error: identifier \"None\" is undefined\r\n\r\nbecause the code in question contains the statement\r\n\r\n #define THREADS None\r\n\r\n***UPDATE: Batch size reduction as suggested by Sudeep above looks promising - it's training and through 53 batches.***\r\n", "102009": "@Anil is there a way to fit the localizer using less memory? It exhausts the memory on my 4GB gpu:\r\n\r\n \r\nTraceback (most recent call last):\r\n File \"./localizer.py\", line 77, in\r\n callbacks=callbacks)\r\n File \"/home/jfk/neon/neon/models/model.py\", line 109, in fit\r\n self.initialize(dataset, cost)\r\n File \"/home/jfk/neon/neon/models/model.py\", line 79, in initialize\r\n self.layers.allocate()\r\n File \"/home/jfk/neon/neon/layers/container.py\", line 82, in allocate\r\n l.allocate()\r\n File \"/home/jfk/neon/neon/layers/layer.py\", line 1201, in allocate\r\n super(BatchNorm, self).allocate(shared_outputs)\r\n File \"/home/jfk/neon/neon/layers/layer.py\", line 123, in allocate\r\n parallelism=self.parallelism)\r\n File \"/home/jfk/neon/neon/backends/backend.py\", line 516, in iobuf\r\n persist_values=persist_values)\r\n File \"/home/jfk/neon/neon/backends/nervanagpu.py\", line 1026, in zeros\r\n rounding=self.round_mode)._assign(0)\r\n File \"/home/jfk/neon/neon/backends/nervanagpu.py\", line 132, in __init__\r\n self.gpudata = allocator(self.nbytes)\r\npycuda._driver.MemoryError: cuMemAlloc failed: out of memory\r\n \r\n", "101968": "[quote=James King;101967]\r\n\r\n[quote=DataGeek;101965]\r\nIt would be helpful if you can tell how you solved all the problems you had during installation\r\n[/quote]\r\n\r\nThere's not that much to say. After many futile efforts with my existing environment (couldn't find numpy, numpy was the wrong version, pycuda would not install correctly) I decided to create a fresh environment in a VM. This gave me a working install of neon, but then I realized the VM could not see the gpu. So I loaded Ubuntu trusty onto a usb and created a new partition with a completely clean install. Blacklisted nouveau, installed nvidia drivers, and installed cuda per NVIDIA's instructions \r\n\r\nhttp://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/#axzz3ue67q5TI\r\n\r\nThen I followed the instructions on github for installing neon (in a venv, not the sysinstall) and it worked on the first try.\r\n\r\n\r\n[/quote]\r\n\r\nThanks. I think I also need to do the same.", "101966": "[quote=Jesse;101963]\r\n\r\nI'm still unsure of what the input shapes are and if there are any augmentations being used in the neon version, but for anyone who's curious I'm able to get it to learn better (~3.4 on my holdout) if I use batch normalization and adadelta on a smaller network (1/4th the number of filters in each layer). Before I was using Nesterov SGD, so it seems batch norm and adadelta are necessary to some extent.\r\n\r\n[quote=Jesse;101657]\r\n\r\nCould someone explain what the input to the classifier looks like (shape, etc.)? The train crops ...\r\n\r\n[/quote]\r\n\r\n\r\n[/quote]\r\nIt would be nice if you can post some code :)", "101964": "[quote=Anil Thomas;101795]\r\n\r\n[quote]\r\n\r\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon...\r\n\r\n[/quote]\r\n\r\nJames, is this python3? Neon currently needs 2.7.\r\n\r\nThere's some [anaconda specific instructions][1] in the docs that may be helpful.\r\n\r\n\r\n [1]: http://neon.nervanasys.com/docs/latest/user_guide.html#anaconda\r\n\r\n[/quote]\r\n\r\nI was finally able to get neon to build on a fresh install of Trusty (and confirmed it is using the gpu).", "101936": "[quote=Anil Thomas;101795]\r\n\r\n[quote]\r\n\r\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon:\r\n\r\n RuntimeError: module compiled against API version a but this version of numpy is 9\r\n terminate called after throwing an instance of 'std::runtime_error'\r\n what(): numpy failed to initialize\r\n Aborted (core dumped)\r\n Makefile:151: recipe for target 'kernels' failed\r\n make: *** [kernels] Error 134\r\n\r\nThis is Ubuntu 15.10\r\n\r\n[/quote]\r\n\r\nJames, is this python3? Neon currently needs 2.7.\r\n\r\nThere's some [anaconda specific instructions][1] in the docs that may be helpful.\r\n\r\n\r\n [1]: http://neon.nervanasys.com/docs/latest/user_guide.html#anaconda\r\n\r\n[/quote]\r\n\r\nIt's python 2.7. make sysinstall fails with the same error; this is using /usr/bin/python2.7 (trying it without anaconda).\r\n", "101795": "[quote]\r\n\r\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon:\r\n\r\n RuntimeError: module compiled against API version a but this version of numpy is 9\r\n terminate called after throwing an instance of 'std::runtime_error'\r\n what(): numpy failed to initialize\r\n Aborted (core dumped)\r\n Makefile:151: recipe for target 'kernels' failed\r\n make: *** [kernels] Error 134\r\n\r\nThis is Ubuntu 15.10\r\n\r\n[/quote]\r\n\r\nJames, is this python3? Neon currently needs 2.7.\r\n\r\nThere's some [anaconda specific instructions][1] in the docs that may be helpful.\r\n\r\n\r\n [1]: http://neon.nervanasys.com/docs/latest/user_guide.html#anaconda", "101788": "[quote=Anil Thomas;101381]\r\n\r\n[quote]\r\n\r\nAnil, is it correct that Neon cannot be installed if running Anaconda due to the use of a virtualenv?\r\n\r\n[/quote]\r\n\r\nLet me know if you are seeing a specific issue. If virtualenv doesn't work, there's also the system-wide install option:\r\n\r\n cd neon\r\n make sysinstall \r\n\r\n[/quote]\r\n\r\nRegardless of the install method (virtualenv, sysinstall, Anaconda or generic python, I get this error when trying to make neon:\r\n\r\n RuntimeError: module compiled against API version a but this version of numpy is 9\r\n terminate called after throwing an instance of 'std::runtime_error'\r\n what(): numpy failed to initialize\r\n Aborted (core dumped)\r\n Makefile:151: recipe for target 'kernels' failed\r\n make: *** [kernels] Error 134\r\n\r\nThis is Ubuntu 15.10", "101690": "@Anil, Even i got the same error \"AttributeError: 'LocalizerLoader' . There were no issues while sys-wide install (make sysinstall) and i did a 'make' in data/loadeer folder as well", "101652": "Abhishek could you illustrate a bit on the problem regarding \"AttributeError: 'LocalizerLoader' object has no attribute 'loaderlib'\" ?\r\n\r\nI went into the same problem too, and I don't have nvcc on my server. ", "101385": "I think make got interrupted at some point. I have solved this issue :) Thanks!", "101384": "[quote]\r\n\r\nDid you also get this error:\r\n\r\n> 2015-12-15 04:04:03,589 - neon.data.imageloader - ERROR - Unable to\r\n> load loader.so.\r\n\r\n[/quote]\r\n\r\nAbhishek, if you have nvcc in the path, *make* will attempt to build loader.so using nvcc. If not, it will try gcc (in the latter case, you won't be able to use any GPU features).\r\n\r\nGive this a try to narrow down the issue:\r\n\r\n cd neon/data/loader\r\n make\r\n", "101381": "[quote]\r\n\r\nAnil, is it correct that Neon cannot be installed if running Anaconda due to the use of a virtualenv?\r\n\r\n[/quote]\r\n\r\nLet me know if you are seeing a specific issue. If virtualenv doesn't work, there's also the system-wide install option:\r\n\r\n cd neon\r\n make sysinstall ", "101375": "[quote=Sudeep Juvekar;101368]\r\n\r\nThat shouldn't happen. Did your 'make' run successfully? It should create .so in neon/data/loader/. Did you activate venv in neon?\r\n\r\nProblem with me is that the cuda kernel fails to compile with Kepler in localize.py\r\n\r\nIt seems like they used to have a support for older GPUs in an earlier version (0.9.0), but it was deprecated. Try running 'make -e GPU=cudanet' (I doubt it will work with latest commits though). Just in case it succeeds, please report back :D\r\n\r\n[/quote]\r\n\r\nNo. It doesnt. On CPU, it will take 15 hours, lol ( for one epoch :( )\r\n", "101368": "That shouldn't happen. Did your 'make' run successfully? It should create .so in neon/data/loader/. Did you activate venv in neon?\r\n\r\nProblem with me is that the cuda kernel fails to compile with Kepler in localize.py\r\n\r\nIt seems like they used to have a support for older GPUs in an earlier version (0.9.0), but it was deprecated. Try running 'make -e GPU=cudanet' (I doubt it will work with latest commits though). Just in case it succeeds, please report back :D", "101347": "[quote=James King;101344]\r\n\r\nAnil, is it correct that Neon cannot be installed if running Anaconda due to the use of a virtualenv?\r\n\r\n[/quote]\r\n\r\nI don't think so. Go to folder anaconda -> lib -> python-2.7 -> site-packages -> git clone https://github.com/NervanaSystems/neon\r\n\r\ncd neon \r\n\r\npython setupy.py install\r\n\r\nwill be installed in few seconds\r\n\r\n*I might be wrong but I have used the same method to install neon and used in other competition before. They might have made some changes in neon repo.", "101344": "Anil, is it correct that Neon cannot be installed if running Anaconda due to the use of a virtualenv?", "101204": "[quote]\r\n\r\nI see actually at the first position of the leaderboard there is another guy also from Nervana, do you use similar approaches or his method is not related to yours?\r\n\r\n[/quote]\r\n\r\nHe used the exact same code that is published on github (with the bug and all). Dr. Luke is just luckier than you and me ;-)\r\n", "101165": "[quote=Anil Thomas;101115]\r\n\r\nHold on! Found a bug in the code. It turns out that the probability of getting a 3.5ish score in a single try is less than 0.25. Will push an update tomorrow that should make it more reliable.\r\n\r\n[/quote]\r\n\r\n\r\nThank you for sharing, your code looks neat, waiting for your bug fix. We just ran into this competiton, so your work could be a good start point.\r\n\r\nI see actually at the first position of the leaderboard there is another guy also from Nervana, do you use similar approaches or his method is not related to yours?", "101140": "righto, I ran the code twice. On both occasions, the encoder stopped early at ~10 epochs, and produced pretty bad crops, probably 5% of the crops look reasonable. The classifier produced a test score of ~6.6.\r\n\r\n[quote=Anil Thomas;101115]\r\n\r\nHold on! Found a bug in the code. It turns out that the probability of getting a 3.5ish score in a single try is less than 0.25. Will push an update tomorrow that should make it more reliable.\r\n\r\n[/quote]\r\n", "100913": "Cool, now everyone's using Neon for this. Such an effective way to gain attention in a crowded space of deep learning tools... \r\n\r\nbut I think it's well worth it. Thanks for the generous sharing! Will be the 1st time to try Neon, but it seems cool.\r\n\r\n[quote=Anil Thomas;100902]\r\n\r\nI have been threatening to release the source code for a while. [Here][1] it is, finally! With 4 weeks left before the deadline, there's hopefully enough time for everyone to process the code and make improvements.\r\n\r\nAs given, it should reproduce my current score (which is at second place on the leaderboard). Note that you will see quite a bit of variation from run to run, so it could take multiple tries to get a good score.\r\n\r\n\r\n [1]: https://github.com/anlthms/whale-2015\r\n\r\n[/quote]\r\n", "100847": "[quote=Schurik;100633]\r\n\r\nThanks @Anil Thomas for sharing your code. Could you tell me how to load my croped images into neon or which example is best for loading own imageset.\r\n\r\n[/quote]\r\n\r\nThe [alexnet example][1] is a good place to start. It requires that you pre-process the images with the provided [batch writer utility][2].\r\n\r\n\r\n [1]: https://github.com/NervanaSystems/neon/blob/master/examples/alexnet.py\r\n [2]: https://github.com/NervanaSystems/neon/blob/master/neon/util/batch_writer.py", "100633": "Thanks @Anil Thomas for sharing your code. Could you tell me how to load my croped images into neon or which example is best for loading own imageset.", "100618": "Thanks @Anil Thomas for sharing this. Look forward to testing the code.\r\n", "100245": "[quote=rc;99530]\r\n\r\nDo you train two separate networks to detect the two points or in a double-headed network? \r\n[/quote]\r\n\r\nI trained two separate networks.\r\n\r\n[quote]\r\nIs the mask a circle or a an ellipse centered at the point? How large should they be?\r\n[/quote]\r\n\r\nI draw a circle (a few pixels wide) around the point of interest. Instead of setting those pixels to white, I set the intensities correlated to the proximity to the point of interest.\r\n\r\nI will post a video recording of the entire meetup. That should make this somewhat clearer.\r\n", "100244": "[quote=dietCoke;99452]\r\n\r\nCan you give a sense for how accurate your autoencoder is? \r\n\r\n[/quote]\r\n\r\nThe crops are of varying quality. Most of them look reasonable. But there are many that are completely off. Here's a sample (also in the slide deck).\r\n![cropped images][1]\r\n\r\nThis was good enough for a logloss of around 4.\r\n\r\n\r\n [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/100244/3355/crops.png?sv=2012-02-12&se=2015-12-07T05%3A53%3A56Z&sr=b&sp=r&sig=8vrWDCzl%2FigvULzzbfldmWNlFefnvOKMkg%2BGhNm1fbw%3D", "99530": "Thanks a lot for sharing!\r\n\r\nDo you train two separate networks to detect the two points or in a double-headed network? Multi-task training usually performs better since multiple supervision signals are stronger than a single signal. If the latter is the case, which layer should the two heads branch?\r\n\r\nThe slide presents two training methods with co-ordinates and with a mask. The SumSquared cost function used in the detection network indicates that the label actually used is the mask. How to generate the segmentation mask from the point coordinate? Is the mask a circle or a an ellipse centered at the point? How large should they be?", "99452": "Can you give a sense for how accurate your autoencoder is? \r\n\r\nAlso, thanks for the code!", "99439": "Do we have permission to use the bonnet-tip and blowhead (point1 and point2) json files?", "101662": "" } }
![\"cropped](\"https://storage.googleapis.com/kaggle-forum-message-attachments/100244/3355/crops.png?sv=2012-02-12&se=2015-12-07T05:53:56Z&sr=b&sp=r&sig=8vrWDCzl%2FigvULzzbfldmWNlFefnvOKMkg%2BGhNm1fbw%3D\")