Hi All,
\n\nWe just make an end-to-end deep learning tutorial with lb score 0.0392:
\n\nhttps://github.com/dmlc/mxnet/tree/master/example/kaggle-ndsb2
\n\nNotice this is a very simple model with no attempt to optimize the structure or hyper parameters, you can build fantastic network based on it. While this tutorial is written in python, MXNet comes with support for other popular languages such as R and Julia which can also be used. You are more than welcomed to try and contribute back to this example.
\n\nBests,
\n\nBing
", "messages": [ { "id": 102412, "postDate": "2015-12-22T11:52:46.003Z", "content": "Hi All,
\n\nWe just make an end-to-end deep learning tutorial with lb score 0.0392:
\n\nhttps://github.com/dmlc/mxnet/tree/master/example/kaggle-ndsb2
\n\nNotice this is a very simple model with no attempt to optimize the structure or hyper parameters, you can build fantastic network based on it. While this tutorial is written in python, MXNet comes with support for other popular languages such as R and Julia which can also be used. You are more than welcomed to try and contribute back to this example.
\n\nBests,
\n\nBing
", "rawMarkdown": "Hi All,\r\n\r\nWe just make an end-to-end deep learning tutorial with lb score 0.0392:\r\n\r\nhttps://github.com/dmlc/mxnet/tree/master/example/kaggle-ndsb2\r\n\r\n\r\nNotice this is a very simple model with no attempt to optimize the structure or hyper parameters, you can build fantastic network based on it. While this tutorial is written in python, MXNet comes with support for other popular languages such as R and Julia which can also be used. You are more than welcomed to try and contribute back to this example.\r\n\r\nBests,\r\n\r\nBing", "votes": 63 }, { "id": 103814, "postDate": "2016-01-06T18:32:27.367Z", "content": "@EIGSI, I don't want to give away too many secrets, but here are some general suggestions:
\n\nNumbers 1 and 2 are pretty standard for neural nets and I nearly always use them. Number 3 is always worth trying, it's fairly quick to try and can sometimes have a large payoff.
\n\nI wouldn't worry about learning rate or momentum tuning till you fix your overfitting issues. All you would likely be able to do is to increase the amount of overfit. Averaging the results of cross validation is a perfectly reasonable idea, but likely very, very slow. It's not something I'd try till I'd completely run out of other ideas.
\n\nGood luck!
", "rawMarkdown": "@EIGSI, I don't want to give away too many secrets, but here are some general suggestions:\r\n\r\n1. With validation curves like these, you should definitely look into early stopping if you aren't using it already! You appear to be overfitting pretty seriously.\r\n2. Look into data augmentation. Did I mention that you seem to be overfitting ;-)\r\n3. Look at ways to improve the preprocessing code. \r\n\r\nNumbers 1 and 2 are pretty standard for neural nets and I nearly always use them. Number 3 is always worth trying, it's fairly quick to try and can sometimes have a large payoff. \r\n\r\nI wouldn't worry about learning rate or momentum tuning till you fix your overfitting issues. All you would likely be able to do is to increase the amount of overfit. Averaging the results of cross validation is a perfectly reasonable idea, but likely very, very slow. It's not something I'd try till I'd completely run out of other ideas.\r\n\r\nGood luck!\r\n", "votes": 9 }, { "id": 102448, "postDate": "2015-12-22T18:11:07.710Z", "content": "thumbs up for MXnet!
\n\nFor people who want to install GPU version of MXnet, please refer to this blog https://no2147483647.wordpress.com/2015/12/07/deep-learning-for-hackers-with-mxnet-1/
\n\nPlus a fun of neural art with MXnet: https://no2147483647.wordpress.com/2015/12/21/deep-learning-for-hackers-with-mxnet-2/
", "rawMarkdown": "thumbs up for MXnet!\r\n\r\nFor people who want to install GPU version of MXnet, please refer to this blog https://no2147483647.wordpress.com/2015/12/07/deep-learning-for-hackers-with-mxnet-1/\r\n\r\nPlus a fun of neural art with MXnet: https://no2147483647.wordpress.com/2015/12/21/deep-learning-for-hackers-with-mxnet-2/", "votes": 8 }, { "id": 103815, "postDate": "2016-01-06T18:46:25.320Z", "content": "@WD you are welcome. Yes, AWS GPU instance needs some kernel image update for installing CUDA if ones wants to install from Ubuntu scratch (https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN) ) I will make a MXnet version for it. Or, one can use docker anyway http://mxnt.ml/en/latest/build.html#docker-images
\n\nAnd, 'star-ed' means 'click the star button on the right top and add a star to MXnet github' :-) I am wondering why the slow tensorflow has 10k+ stars on github while all core code pull requests being rejected (you call it open source?), but the faster, easy-to-use, real open source MXnet only has 2000+ stars now. Please click the star button https://github.com/dmlc/mxnet
\n\nThe reasons why MXnet is good for competitions:
\n\n@EIGSI some of my tricks/techniques:
\n\nThank you so much for sharing (and advancing mxnet)!\nI had a small bug which is in the line
\n\n img = preproc(f.pixel_array / np.max(f.pixel_array))\nThe problem is that for me, dicom, returns int pixels so dividing by the max returns an all zero image (except for the max) so a possible fix is:
img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\nThe easiest should use SFrame.\nI will write a tutorial on how to use SFrame with MXNet. Hopefully in this week.\nhttps://github.com/dmlc/mxnet/tree/master/plugin/sframe
\n\nhttps://github.com/dato-code/SFrame
\n\n[quote=WD;105654]
\n\n@phunter. thanks for this. i will look into both the CSV route and the Io.record route. in the CSV approach - i guess the easiest way would be to work with the numpy "img" object - no? i will play around with this and make code public if i find something useful to others
\n\n f = dicom.read_file(path)\n img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\n dst_path = path.rsplit(".", 1)[0] + ".64x64.jpg" #create jpg out of all files\n[/quote]
", "rawMarkdown": "The easiest should use SFrame.\r\nI will write a tutorial on how to use SFrame with MXNet. Hopefully in this week.\r\nhttps://github.com/dmlc/mxnet/tree/master/plugin/sframe\r\n\r\nhttps://github.com/dato-code/SFrame\r\n\r\n\r\n[quote=WD;105654]\r\n\r\n@phunter. thanks for this. i will look into both the CSV route and the Io.record route. in the CSV approach - i guess the easiest way would be to work with the numpy \"img\" object - no? i will play around with this and make code public if i find something useful to others\r\n\r\n f = dicom.read_file(path)\r\n img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\r\n dst_path = path.rsplit(\".\", 1)[0] + \".64x64.jpg\" #create jpg out of all files\r\n\r\n[/quote]", "votes": 4 }, { "id": 106773, "postDate": "2016-02-03T20:58:14.067Z", "content": "[quote=WD;106742]
\n\n@Franc. thanks so much. should one add as well a convolutional relu layer between the flatten and the fc1? as per below? or is that not correct / needed?
\n\n flatten = mx.symbol.Flatten(net)\n flatten = mx.symbol.Dropout(flatten)\n flatten = mx.sym.Activation(flatten, act_type="relu") ###to be checked\n fc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=1024)\n fc1 = mx.sym.Activation(fc1, act_type="relu")\n fc1 = mx.symbol.Dropout(fc1)\n fc2 = mx.symbol.FullyConnected(data=fc1, num_hidden=1)\n[/quote]\nNo, usually you do not put convolutinal layer between flatten and FC layer. Notice, that in the third line in the code above you have only activation layer. I would simply remove the third line of code, i.e. flatten = mx.sym.Activation(flatten, act_type="relu")
I just want to stick this out here for those still stuck trying to process on Windows with Python 3.5. The issue with the processing file failing at on the last line is because when using csv.writer on a Windows machine it will add a extra line to each row of data. To fix this error open the file that you want to write to with open(file, "w", newline = ''). Many thanks to Scott Smith for telling me this almost of month ago. Sad thing is I just realized it today after I fixed the code.
", "rawMarkdown": "I just want to stick this out here for those still stuck trying to process on Windows with Python 3.5. The issue with the processing file failing at on the last line is because when using csv.writer on a Windows machine it will add a extra line to each row of data. To fix this error open the file that you want to write to with open(file, \"w\", newline = ''). Many thanks to Scott Smith for telling me this almost of month ago. Sad thing is I just realized it today after I fixed the code. ", "votes": 1 }, { "id": 106343, "postDate": "2016-01-30T13:10:15.347Z", "content": "@Linear regression. Tooled the Mxnet tutoral over from logistic to linear regression. happy to share code if of interest to others
", "rawMarkdown": "@Linear regression. Tooled the Mxnet tutoral over from logistic to linear regression. happy to share code if of interest to others ", "votes": 1 }, { "id": 106251, "postDate": "2016-01-29T12:29:29.980Z", "content": "@Tim. At a more elementary level, i was thinking, that if the network outputs (on e.g. a hypothetical 8 output notes, labelled from 1 to 8): 0.1, 0.1, 0.1, 0.1, 0.99, 0.99, 0.99, 0.99.
\n\nThen the Mxnet would create a CDF that would be 0.1, 0.1, 0.1, 0.1, 0.99, 0.99, 0.99, 0.99. This is almost a step function at node 5.
\n\nHowever, as the network is equally confident that the output might be 5,6,7 and 8, a naieve best guess on best estimator would potentially be the average between 5 to 8, e.g. 6.5
\n\n(normalizing the inputs and subsequently calculating a CDF of e.g. (0.025,0.05,0.075,0.1,0.325,0.55,0.775,1) also does intuitively not make sense)
\n\nHence - I was wondering whether there is an opportunity / need to adjust the CDF that the Mxnet tutoral calculates? I realize that this is a trivial example - but still would love to get your thoughts.
", "rawMarkdown": "@Tim. At a more elementary level, i was thinking, that if the network outputs (on e.g. a hypothetical 8 output notes, labelled from 1 to 8): 0.1, 0.1, 0.1, 0.1, 0.99, 0.99, 0.99, 0.99.\r\n\r\nThen the Mxnet would create a CDF that would be 0.1, 0.1, 0.1, 0.1, 0.99, 0.99, 0.99, 0.99. This is almost a step function at node 5. \r\n\r\nHowever, as the network is equally confident that the output might be 5,6,7 and 8, a naieve best guess on best estimator would potentially be the average between 5 to 8, e.g. 6.5 \r\n\r\n(normalizing the inputs and subsequently calculating a CDF of e.g. (0.025,0.05,0.075,0.1,0.325,0.55,0.775,1) also does intuitively not make sense) \r\n\r\nHence - I was wondering whether there is an opportunity / need to adjust the CDF that the Mxnet tutoral calculates? I realize that this is a trivial example - but still would love to get your thoughts. \r\n", "votes": 1 }, { "id": 106079, "postDate": "2016-01-28T10:54:50.850Z", "content": "@Bartek that's the beauty of neural networks they capture the "je ne sais quoi" (I DONT KNOW WHAT) in the data :). What makes you able to distinguish faces of 2 different people, you can't really explain why? There are some obvious things you see but there are latent features baked into your brain that make you able to distinguish faces. That's what neural networks capture. The technical explanation is that they capture hierarchical representations of features in the data. In my opinion the first tutorial is trying to explain why 2 faces are different (segmenting e.t.c) there are just too many variables, scale, noise, orientation e.t.c for it to do a good job. Early in the competition people who tried that approach reported scores of 0.1xxxx which is very bad compared to the current scores on the leaderboard.
", "rawMarkdown": "@Bartek that's the beauty of neural networks they capture the \"je ne sais quoi\" (I DONT KNOW WHAT) in the data :). What makes you able to distinguish faces of 2 different people, you can't really explain why? There are some obvious things you see but there are latent features baked into your brain that make you able to distinguish faces. That's what neural networks capture. The technical explanation is that they capture hierarchical representations of features in the data. In my opinion the first tutorial is trying to explain why 2 faces are different (segmenting e.t.c) there are just too many variables, scale, noise, orientation e.t.c for it to do a good job. Early in the competition people who tried that approach reported scores of 0.1xxxx which is very bad compared to the current scores on the leaderboard.", "votes": 1 }, { "id": 106024, "postDate": "2016-01-27T23:16:26.087Z", "content": "@WD, that's an interesting point. If my quick scribblings are correct, then computing the MSE on the CDF (versus a step function) is equivalent to computing a weighted MSE on the PDF. The weights would be linearly decreasing[1]. So, for instance the weights could be 600, 599, ...., 2, 1. Thus, if one is outputting the CDF directly one is putting more emphasis on the lower values than upper values. This makes intuitive sense, since an incorrect early value in the PDF will result in all subsequent values of the CDF being off.
\n\nI would suspect that computing the CDF directly will result in better scores than computing the PDF (using softmax) and then converting that to CDF since CDF is the eventual target. I may be wrong about that though, so I encourage you to try it.
\n\n[EDIT] OK, I looked at it a bit more and that math is wrong, but I think the conclusion is probably more or less right, earlier points end up being weighted more than later points.
\n\n[1] Whether the weights are linear or quadratic depends on whether one is applying them before or after squaring. Here I assume that they are applied before squaring.
", "rawMarkdown": "@WD, that's an interesting point. If my quick scribblings are correct, then computing the MSE on the CDF (versus a step function) is equivalent to computing a weighted MSE on the PDF. The weights would be linearly decreasing[1]. So, for instance the weights could be 600, 599, ...., 2, 1. Thus, if one is outputting the CDF directly one is putting more emphasis on the lower values than upper values. This makes intuitive sense, since an incorrect early value in the PDF will result in all subsequent values of the CDF being off. \r\n\r\nI would suspect that computing the CDF directly will result in better scores than computing the PDF (using softmax) and then converting that to CDF since CDF is the eventual target. I may be wrong about that though, so I encourage you to try it. \r\n\r\n[EDIT] OK, I looked at it a bit more and that math is wrong, but I think the conclusion is probably more or less right, earlier points end up being weighted more than later points.\r\n\r\n\r\n[1] Whether the weights are linear or quadratic depends on whether one is applying them before or after squaring. Here I assume that they are applied before squaring.", "votes": 1 }, { "id": 106009, "postDate": "2016-01-27T22:07:55.113Z", "content": "@Tim. As a follow-up thought - it seems that the indepedent sigmoid approach and the resulting CDF approximation creates a CDF that is a little "biased to the left". A actual softmax "adjustment" and proper CDF calculation from these values might lead to an improvement in scores. will try this out later.
", "rawMarkdown": "@Tim. As a follow-up thought - it seems that the indepedent sigmoid approach and the resulting CDF approximation creates a CDF that is a little \"biased to the left\". A actual softmax \"adjustment\" and proper CDF calculation from these values might lead to an improvement in scores. will try this out later. ", "votes": 1 }, { "id": 105964, "postDate": "2016-01-27T18:41:00.023Z", "content": "@WD, even though this layer is labelled "softmax", I believe it's just a sigmoid output. The net is supposed to figure out that the output should look like a CDF during training, so the output should be approximately ascending, but it can have glitches. That is why there is submission_helper, it fixes up any non-monotonicity.
Hope that helps.
", "rawMarkdown": "\r\n@WD, even though this layer is *labelled* \"softmax\", I believe it's just a sigmoid output. The net is supposed to figure out that the output should look like a CDF during training, so the output should be approximately ascending, but it can have glitches. That is why there is `submission_helper`, it fixes up any non-monotonicity.\r\n\r\nHope that helps.", "votes": 1 }, { "id": 105654, "postDate": "2016-01-25T16:46:54.887Z", "content": "@phunter. thanks for this. i will look into both the CSV route and the Io.record route. in the CSV approach - i guess the easiest way would be to work with the numpy "img" object - no? i will play around with this and make code public if i find something useful to others
\n\n f = dicom.read_file(path)\n img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\n dst_path = path.rsplit(".", 1)[0] + ".64x64.jpg" #create jpg out of all files\nThere is rotation and a bunch of augmentation method, but require ImageRecord IO. For this example, we only want to show how to use MXNet so we choose CSV.
\n\nhttps://mxnet.readthedocs.org/en/latest/python/io.html#mxnet.io.ImageRecordIter
\n\n[quote=phunter;105650]
\n\nthe mxnet tutorial code has not (yet) support image rotation. one can modify preprocessing.py and generate CSV with rotation by scikit-image
\n\n[/quote]
", "rawMarkdown": "There is rotation and a bunch of augmentation method, but require ImageRecord IO. For this example, we only want to show how to use MXNet so we choose CSV.\r\n\r\nhttps://mxnet.readthedocs.org/en/latest/python/io.html#mxnet.io.ImageRecordIter\r\n\r\n\r\n[quote=phunter;105650]\r\n\r\nthe mxnet tutorial code has not (yet) support image rotation. one can modify preprocessing.py and generate CSV with rotation by scikit-image\r\n\r\n[/quote]", "votes": 1 }, { "id": 105164, "postDate": "2016-01-20T15:34:57.017Z", "content": "G2x2 spot runs at around 0.1$ per hour. Regular instances (not spot) are more expensive - 0.65$. it depends on your launch zone. The price graphs can be found on AWS - when go to spot instances.
", "rawMarkdown": "G2x2 spot runs at around 0.1$ per hour. Regular instances (not spot) are more expensive - 0.65$. it depends on your launch zone. The price graphs can be found on AWS - when go to spot instances.\r\n", "votes": 1 }, { "id": 105066, "postDate": "2016-01-19T18:09:39.490Z", "content": "I am finishing up code to extract and plot training and validation curves, and I will share this code today or tomorrow. I had a quick question. If one uses a terminal mode to access one's AWS instance (e.g. using Putty) and either runs scripts from the bash command line or from an ipython shell, then is it possible to plot graphs visually? or does one either a) have to install an ipython notebook, or b) save-to-image, transfer the files back to one's local computer, and plot locally?
", "rawMarkdown": "I am finishing up code to extract and plot training and validation curves, and I will share this code today or tomorrow. I had a quick question. If one uses a terminal mode to access one's AWS instance (e.g. using Putty) and either runs scripts from the bash command line or from an ipython shell, then is it possible to plot graphs visually? or does one either a) have to install an ipython notebook, or b) save-to-image, transfer the files back to one's local computer, and plot locally? ", "votes": 1 }, { "id": 105013, "postDate": "2016-01-18T20:33:18.983Z", "content": "i am trying to visualize the network. Some others might be interested in this as well. I thought the code below should do the trick. However, even doing all the sudo apt-get installs (ubuntu) i still get a graphviz library not found error. let me know if anyone has experience with this or has encountered the same issue or knows how to resolve this!
\n\n\n", "rawMarkdown": "i am trying to visualize the network. Some others might be interested in this as well. I thought the code below should do the trick. However, even doing all the sudo apt-get installs (ubuntu) i still get a graphviz library not found error. let me know if anyone has experience with this or has encountered the same issue or knows how to resolve this! \r\n\r\n\r\n> import graphviz import find_mxnet import mxnet as mx import importlib\r\n> \r\n> def get_lenet():\r\n> #the code from the train.py file \r\n\r\n> network = get_lenet()\r\n> \r\n> mx.viz.plot_network(network)\r\n\r\n", "votes": 1 }, { "id": 104979, "postDate": "2016-01-18T15:06:23.807Z", "content": "import graphviz import find_mxnet import mxnet as mx import importlib
\n \ndef get_lenet():\n #the code from the train.py file
\n \nnetwork = get_lenet()
\n \nmx.viz.plot_network(network)
\n
@EIGSI - you can also log the validation curves by using e.g. stytole_model.fit(X=data_train, eval_metric = mx.metric.np(CRPS),batch_end_callback = mx.callback.log_train_metric(32))
", "rawMarkdown": "@EIGSI - you can also log the validation curves by using e.g. stytole_model.fit(X=data_train, eval_metric = mx.metric.np(CRPS),batch_end_callback = mx.callback.log_train_metric(32))\r\n", "votes": 1 }, { "id": 104955, "postDate": "2016-01-18T08:47:26.240Z", "content": "@Jon Carlies: since precompiled mxnet libraries are only cpu enabled, replace all : devs = [mx.gpu(0)] with devs = [mx.cpu()].
[quote=Scott Smith;104308]
\n\n[quote=Matthew Tubs;104174]
\n\nUpdate: I got the code running smoother now. Turned out that while messing with R I made images of all the dicom files and that was interrupting the code.
\n\nHowever, I have ran into a different error at the end of the run. Can anyone help me figure why this index went out of range?
\n\nif split_to_train[cnt]:
\n\nIndexError: list index out of range
\n\n[/quote]
\n\n@Matthew: I think I ran into the same issues preprocessing with Python 3.5 on Windows.
\n\n[/quote]
\n\nI'm on the last line of code and reviewing #2 in this post to try and solve my index issue. For me, he train_frames has a length of 5,293 yet train-64x64-data.csv has double that which I believe is why there is an index error. Am I understanding this correctly...how many rows should I have in train-64x64-data.csv?
", "rawMarkdown": "[quote=Scott Smith;104308]\r\n\r\n[quote=Matthew Tubs;104174]\r\n\r\nUpdate: I got the code running smoother now. Turned out that while messing with R I made images of all the dicom files and that was interrupting the code. \r\n\r\nHowever, I have ran into a different error at the end of the run. Can anyone help me figure why this index went out of range?\r\n\r\n if split_to_train[cnt]:\r\n\r\nIndexError: list index out of range\r\n\r\n[/quote]\r\n\r\n@Matthew: I think I ran into the same issues preprocessing with Python 3.5 on Windows.\r\n\r\n 1. In the get_frames() function, the second time you run there are no frames detected due to the presence of jpg images introduced. The adjustment in [@Woolsey's comment][1] corrects this by limiting the 'files' object to only dcm entries.\r\n 2. With Python 3 and Windows, there can be line returns introduced where you don't want them. Check your train-label.csv file. I had additional lines introduced in the get_label_map() function that would lead to empty rows in train_label.csv. I think I fixed this by using 'fi = open(fname, 'r', newline ='')' in get_label_map(), but the file is not on this computer. It might help to try preprocessing with the first 5 train and validation studies only and then take a look at the csv's. If there are extra lines or empty lines at the end of the files, this will be a problem when you process your larger set.\r\n\r\n [1]: https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/18079/end-to-end-deep-learning-tutorial-0-0392/103854#post103854\r\n\r\n[/quote]\r\n\r\nI'm on the last line of code and reviewing #2 in this post to try and solve my index issue. For me, he train_frames has a length of 5,293 yet train-64x64-data.csv has double that which I believe is why there is an index error. Am I understanding this correctly...how many rows should I have in train-64x64-data.csv?\r\n", "votes": 1 }, { "id": 104308, "postDate": "2016-01-11T16:16:21.163Z", "content": "[quote=Matthew Tubs;104174]
\n\nUpdate: I got the code running smoother now. Turned out that while messing with R I made images of all the dicom files and that was interrupting the code.
\n\nHowever, I have ran into a different error at the end of the run. Can anyone help me figure why this index went out of range?
\n\nif split_to_train[cnt]:
\n\nIndexError: list index out of range
\n\n[/quote]
\n\n@Matthew: I think I ran into the same issues preprocessing with Python 3.5 on Windows.
\n\n[quote=Matthew Tubs;104166]
\n\n[quote=earino;104164]
\n\n[quote=Matthew Tubs;104163]\nFranc
\n\nI've been trying to get the Preprocessing.py file you attached running and I've ran into a issue. Would you help me out in understanding what is going on? I am running Python 3.5 on windows, and I am receiving the following error.
\n\ntran_index = np.loadtxt("./tran_label.csv", delimiter=",")[:,0].astype("int")
\n\nIndexError: too many indices for array
\n\nThanks much\n [/quote]
\n\nis that tran_label supposed to be train_label?
\n\n[/quote]\nI did a replace all to see if the problem was something attached to the name of the file.
\n\n[/quote]
\n\nUpdate: I got the code running smoother now. Turned out that while messing with R I made images of all the dicom files and that was interrupting the code.
\n\nHowever, I have ran into a different error at the end of the run. Can anyone help me figure why this index went out of range?
\n\nif split_to_train[cnt]:
\n\nIndexError: list index out of range
", "rawMarkdown": "[quote=Matthew Tubs;104166]\r\n\r\n[quote=earino;104164]\r\n\r\n[quote=Matthew Tubs;104163]\r\nFranc \r\n\r\nI've been trying to get the Preprocessing.py file you attached running and I've ran into a issue. Would you help me out in understanding what is going on? I am running Python 3.5 on windows, and I am receiving the following error.\r\n\r\n tran_index = np.loadtxt(\"./tran_label.csv\", delimiter=\",\")[:,0].astype(\"int\")\r\n\r\nIndexError: too many indices for array\r\n\r\nThanks much\r\n [/quote]\r\n\r\nis that tran_label supposed to be train_label?\r\n\r\n\r\n[/quote]\r\nI did a replace all to see if the problem was something attached to the name of the file.\r\n\r\n[/quote]\r\n\r\nUpdate: I got the code running smoother now. Turned out that while messing with R I made images of all the dicom files and that was interrupting the code. \r\n\r\nHowever, I have ran into a different error at the end of the run. Can anyone help me figure why this index went out of range?\r\n\r\n if split_to_train[cnt]:\r\n\r\nIndexError: list index out of range", "votes": 1 }, { "id": 104116, "postDate": "2016-01-09T05:38:46.777Z", "content": "Hi! What is the license of your tutorial? I don't see one posted in the github?
", "rawMarkdown": "Hi! What is the license of your tutorial? I don't see one posted in the github?", "votes": 1 }, { "id": 104026, "postDate": "2016-01-08T16:27:52.357Z", "content": "i received an odd error running processing.py. what might be the root-cause?
\n\nW
\n\n800 slices processed\nTraceback (most recent call last):\n File "Preprocessing.py", line 134, in \n valid_lst = write_data_csv("./validate-64x64-data.csv", validate_frames, lambda x: crop_resize(x, 64))\n File "Preprocessing.py", line 66, in write_data_csv\n img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\n File "/usr/lib/python2.7/dist-packages/dicom/dataset.py", line 405, in _get_pixel_array\n return self._getPixelArray()\n File "/usr/lib/python2.7/dist-packages/dicom/dataset.py", line 400, in _getPixelArray\n self._PixelArray = self._PixelDataNumpy()\n File "/usr/lib/python2.7/dist-packages/dicom/dataset.py", line 382, in _PixelDataNumpy\n arr = arr.reshape(self.Rows, self.Columns)\nValueError: total size of new array must be unchanged\nubuntu@ip-172-31-7-91:~$
", "rawMarkdown": "i received an odd error running processing.py. what might be the root-cause? \r\n\r\nW\r\n\r\n800 slices processed\r\nTraceback (most recent call last):\r\n File \"Preprocessing.py\", line 134, inThe figure shows my best submission so far using this code (0.038332). I am using the local train/test datasets generated by Preprocessing.py (1/10th for local test, rest is for training). Changed the epoch to 100 for both networks. I also had to reduce batch_size to 8 due to my low-memory GPU.
\n\nFor the sake of learning, what else can we do to improve this code's performance? Playing with the learning rate, momentum? Doing 10-fold cross-validation and averaging predictions of those 10 models?
", "rawMarkdown": "The figure shows my best submission so far using this code (0.038332). I am using the local train/test datasets generated by Preprocessing.py (1/10th for local test, rest is for training). Changed the epoch to 100 for both networks. I also had to reduce batch_size to 8 due to my low-memory GPU.\r\n\r\nFor the sake of learning, what else can we do to improve this code's performance? Playing with the learning rate, momentum? Doing 10-fold cross-validation and averaging predictions of those 10 models? ", "votes": 1 }, { "id": 103672, "postDate": "2016-01-05T12:50:38.567Z", "content": "Train.py code worked so well, but mx.model.FeedForward.create in Train.R code does not work. What is the cause?
\n\n> # Training the stytole net\n> mx.set.seed(0)\n> stytole_model <- mx.model.FeedForward.create(\n+ X = data_train,\n+ ctx = mx.gpu(0),\n+ symbol = network,\n+ num.round = 65,\n+ learning.rate = 0.001,\n+ wd = 0.00001,\n+ momentum = 0.9,\n+ eval.metric = mx.metric.CRPS\n+ )\nStart training with 1 devices\n[1] Train-CRPS=0.248537092806794\n[2] Train-CRPS=0.249575979181468\n[3] Train-CRPS=0.249281826982739\n[4] Train-CRPS=0.248988226596111\n[5] Train-CRPS=0.248694746602638\n[6] Train-CRPS=0.248402110901456\n[7] Train-CRPS=0.248110486441219\n[8] Train-CRPS=0.247818248773282\n[9] Train-CRPS=0.247526859680184\n[10] Train-CRPS=0.247235201653695\n(Omitted)\n[55] Train-CRPS=0.234554654820709\n[56] Train-CRPS=0.234282698395829\n[57] Train-CRPS=0.234008295194647\n[58] Train-CRPS=0.233738730834488\n[59] Train-CRPS=0.233467098573434\n[60] Train-CRPS=0.233195344441995\n[61] Train-CRPS=0.232924924355617\n[62] Train-CRPS=0.232653180647331\n[63] Train-CRPS=0.2323852445735\n[64] Train-CRPS=0.23211576977677\n[65] Train-CRPS=0.231845338545091\n@Bing Xu Thanks for this code and introducing to mxnet!
", "rawMarkdown": "@Bing Xu Thanks for this code and introducing to mxnet!", "votes": 1 }, { "id": 103576, "postDate": "2016-01-04T13:48:07.137Z", "content": "Thank you very much! #2 and #3 helped me to get it to work on my mac.\n[quote=phunter;103532]
\n\n@EIGSI the last epoch takes some memory so it has the risk of failure since 1GB is kind of small. Some suggestions:
\n\nMXNET_BACKWARD_DO_MIRROR=1 python Train.py
\n\n[/quote]
", "rawMarkdown": "Thank you very much! #2 and #3 helped me to get it to work on my mac.\r\n[quote=phunter;103532]\r\n\r\n@EIGSI the last epoch takes some memory so it has the risk of failure since 1GB is kind of small. Some suggestions:\r\n\r\n0. Try Amazon AWS GPU.\r\n1. change to smaller batch size, 16 or less. Do you want to try 8?\r\n2. MXnet has some magic as I mentioned in https://no2147483647.wordpress.com/2015/12/21/deep-learning-for-hackers-with-mxnet-2/ : the latest MXnet support mirror memory which trades off computing time vs memory usage. If you have the latest MXnet, please use:\r\n\r\nMXNET_BACKWARD_DO_MIRROR=1 python Train.py\r\n\r\n[/quote]\r\n", "votes": 1 }, { "id": 103338, "postDate": "2016-01-01T07:35:33.107Z", "content": "@Mathurin:
\n\nI had the same problem on Windows. I fixed it by adding one line of code root=root.replace('\\\\','/') and it works for me. See the relevant part of code below. Hope this will help you.
for root, _, files in os.walk(root_path):\n root=root.replace('\\\\','/')\n if len(files) == 0 or not files[0].endswith(".dcm") or root.find("sax") == -1:\n continue\n prefix = files[0].rsplit('-', 1)[0]\n fileset = set(files)\n expected = ["%s-%04d.dcm" % (prefix, i + 1) for i in range(30)]\n if all(x in fileset for x in expected):\n ret.append([root + "/" + x for x in expected])\n # sort for reproduciblity\n return sorted(ret, key = lambda x: x[0])\n@Jiming Ye, the first two lines convert the pixel values from floating point values in the range [0,1] to a byte value. This conserves a lot of space.
\n\nThe third line computes the differences from frame to frame and uses it instead of the raw frame values. Whether this is useful I can't say.
\n\n[EDIT] Fixed typo where I typed 'frame' instead of difference.
", "rawMarkdown": "@Jiming Ye, the first two lines convert the pixel values from floating point values in the range [0,1] to a byte value. This conserves a lot of space. \r\n\r\nThe third line computes the differences from frame to frame and uses it instead of the raw frame values. Whether this is useful I can't say. \r\n\r\n[EDIT] Fixed typo where I typed 'frame' instead of difference.", "votes": 1 }, { "id": 103030, "postDate": "2015-12-28T13:56:32.767Z", "content": "Thanks for the tutorial. I tried my own approach, but it doesn't work. So I just switched to this tutorial. But it looks really weird for me.
\n\nresized_img *= 255
\n\nreturn resized_img.astype("uint8")
\n\nThen take difference of each frame as input, what would be the data after all these transformations? Why is it useful?
", "rawMarkdown": "Thanks for the tutorial. I tried my own approach, but it doesn't work. So I just switched to this tutorial. But it looks really weird for me. \r\n\r\nresized_img *= 255 \r\n\r\nreturn resized_img.astype(\"uint8\")\r\n\r\nThen take difference of each frame as input, what would be the data after all these transformations? Why is it useful?", "votes": 1 }, { "id": 102786, "postDate": "2015-12-26T01:16:48.217Z", "content": "I'm able to run Preprocessing successfully, but then when trying to Train the data, I get
\n\nTraceback (most recent call last):\n File "Train.py", line 77, in \n network = get_lenet()\n File "Train.py", line 16, in get_lenet\n source = mx.sym.Variable("data")\nAttributeError: 'module' object has no attribute 'sym'
\n\nAnyone have any pointers on this? Thanks.
", "rawMarkdown": "I'm able to run Preprocessing successfully, but then when trying to Train the data, I get \r\n\r\nTraceback (most recent call last):\r\n File \"Train.py\", line 77, inphunter, awesome style tutorial, thank you.
", "rawMarkdown": "phunter, awesome style tutorial, thank you.", "votes": 1 }, { "id": 102716, "postDate": "2015-12-24T23:01:01.570Z", "content": "Sorry prebuild is always late than repo.\nYou may try to build by yourself: https://mxnet.readthedocs.org/en/latest/build.html#building-on-windows
\n\n[quote=BlackCore;102712]
\n\nThank you for sharing the framework & project. I have managed to ran Preprocessing.py, but Train.py failed with the following: AttributeError: module 'mxnet.io' has no attribute 'CSVIter'
\n\nNote that I am using the pre-built Windows version from here: https://github.com/dmlc/mxnet/releases. This is because it takes a while until my NVidia user gets approved. From what I see on https://github.com/dmlc/mxnet, there is a comment: "[IO] Add CSV Iter" from 3 days ago. This means that this change has not been included yet in the pre-built package, and therefore I cannot use mxnet.io.CSVIter present in the Train.py code.
\n\nIs there any way in which I could get a pre-built package with the latest changes?
\n\nThanks!
\n\n[/quote]
", "rawMarkdown": "Sorry prebuild is always late than repo.\r\nYou may try to build by yourself: https://mxnet.readthedocs.org/en/latest/build.html#building-on-windows\r\n\r\n[quote=BlackCore;102712]\r\n\r\nThank you for sharing the framework & project. I have managed to ran Preprocessing.py, but Train.py failed with the following: AttributeError: module 'mxnet.io' has no attribute 'CSVIter'\r\n\r\nNote that I am using the pre-built Windows version from here: https://github.com/dmlc/mxnet/releases. This is because it takes a while until my NVidia user gets approved. From what I see on https://github.com/dmlc/mxnet, there is a comment: \"[IO] Add CSV Iter\" from 3 days ago. This means that this change has not been included yet in the pre-built package, and therefore I cannot use mxnet.io.CSVIter present in the Train.py code. \r\n\r\nIs there any way in which I could get a pre-built package with the latest changes?\r\n\r\nThanks!\r\n\r\n[/quote]\r\n", "votes": 1 }, { "id": 102495, "postDate": "2015-12-23T03:44:40.667Z", "content": "Thx for share.\ndef get_frames(root_path) does not work for windows\nsince os.walk(root_path) will return backslash eg. "/data/train\\493\\study...."\nMaybe replace that with
\n\nroot=root.replace('\\\\','/')\nThanks so much for sharing!
\n\nFrom reading, it looks like this model takes a set of frames from a single slice, performs some preprocessing, predicts the CDF based on that single slice, then accumulates the results from all slices for a given study?
\n\nCheers
", "rawMarkdown": "Thanks so much for sharing!\r\n\r\nFrom reading, it looks like this model takes a set of frames from a single slice, performs some preprocessing, predicts the CDF based on that single slice, then accumulates the results from all slices for a given study?\r\n\r\nCheers", "votes": 1 }, { "id": 102435, "postDate": "2015-12-22T17:00:44.230Z", "content": "I assume you are using Python2? Thanks for pointing it and I will fix it in later today.
\n\n[quote=udibr;102433]
\n\nThank you so much for sharing (and advancing mxnet)!\nI had a small bug which is in the line
\n\n img = preproc(f.pixel_array / np.max(f.pixel_array))\nThe problem is that for me, dicom, returns int pixels so dividing by the max returns an all zero image (except for the max) so a possible fix is:
img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\n[/quote]
", "rawMarkdown": "I assume you are using Python2? Thanks for pointing it and I will fix it in later today.\r\n\r\n[quote=udibr;102433]\r\n\r\nThank you so much for sharing (and advancing mxnet)!\r\nI had a small bug which is in the line \r\n\r\n img = preproc(f.pixel_array / np.max(f.pixel_array))\r\n\r\nThe problem is that for me, `dicom`, returns int pixels so dividing by the max returns an all zero image (except for the max) so a possible fix is:\r\n\r\n img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\r\n\r\n\r\n[/quote]\r\n", "votes": 1 }, { "id": 109395, "postDate": "2016-02-25T18:23:03.380Z", "content": "@Florian, you're welcome. If you want to try this, here is how you implement maximum.accumulate using Theano:
\n\n# Based on examples at http://deeplearning.net/software/theano/library/scan.html\nimport theano\nimport theano.tensor as T\nimport numpy as np\n\nV = T.vector("V")\n\ndef accumulate(val, so_far):\n return T.maximum(val, so_far)\n\noutputs_info = T.as_tensor_variable(np.asarray(0, V.dtype))\ncummax_expr, scan_updates = theano.scan(fn=accumulate,\n outputs_info=outputs_info,\n sequences=V)\ncummax = theano.function(inputs=[V], outputs=cummax_expr)\n\nprint(cummax([0,1,3,2,5,4,6])) # => [ 0. 1. 3. 3. 5. 5. 6.]\nprint(cummax([6,5,4,3,7,4])) # => [ 6. 6. 6. 6. 7. 7.]\nI believe that you'd use cummax_expr in place of your inner loop, and that would probably work (as in compile), but I haven't tried it and Theano frequently surprises me:-) I don't really think it will help, but I've been meaning to try out scan for a while and this was a good excuse. The cummax function does work as a function, but I don't think you want that in your objective function, for that you want a Theano expression instead.
", "rawMarkdown": "@Florian, you're welcome. If you want to try this, here is how you implement maximum.accumulate using Theano:\r\n\r\n # Based on examples at http://deeplearning.net/software/theano/library/scan.html\r\n import theano\r\n import theano.tensor as T\r\n import numpy as np\r\n\r\n V = T.vector(\"V\")\r\n\r\n def accumulate(val, so_far):\r\n return T.maximum(val, so_far)\r\n\r\n outputs_info = T.as_tensor_variable(np.asarray(0, V.dtype))\r\n cummax_expr, scan_updates = theano.scan(fn=accumulate,\r\n outputs_info=outputs_info,\r\n sequences=V)\r\n cummax = theano.function(inputs=[V], outputs=cummax_expr)\r\n\r\n print(cummax([0,1,3,2,5,4,6])) # => [ 0. 1. 3. 3. 5. 5. 6.]\r\n print(cummax([6,5,4,3,7,4])) # => [ 6. 6. 6. 6. 7. 7.]\r\n\r\nI believe that you'd use cummax_expr in place of your inner loop, and that would probably work (as in compile), but I haven't tried it and Theano frequently surprises me:-) I don't really think it will help, but I've been meaning to try out scan for a while and this was a good excuse. The cummax function does work as a function, but I don't think you want that in your objective function, for that you want a Theano expression instead.", "votes": 2 }, { "id": 109382, "postDate": "2016-02-25T16:08:11.683Z", "content": "@Florian, I think you will need to use Theano.scan to implement something like this (you want the equivalent of numpy.maximum.accumulate, and that's implemented using scan in Theano). See http://deeplearning.net/software/theano/library/scan.html.
\n\nHowever, I suggest you not bother with forcing the output to be monotonic in the objective function. Just use MSE and call it a day. When you get the results, then force it to be monotonic, and possibly also when computing the validation score, but I don't think it's worth the trouble as part of the objective.
\n\nHope that helps.
", "rawMarkdown": "\r\n@Florian, I think you will need to use Theano.scan to implement something like this (you want the equivalent of numpy.maximum.accumulate, and that's implemented using scan in Theano). See http://deeplearning.net/software/theano/library/scan.html.\r\n\r\nHowever, I suggest you not bother with forcing the output to be monotonic in the objective function. Just use MSE and call it a day. When you get the results, then force it to be monotonic, and possibly also when computing the validation score, but I don't think it's worth the trouble as part of the objective.\r\n\r\nHope that helps.", "votes": 2 }, { "id": 105977, "postDate": "2016-01-27T19:25:08.497Z", "content": "@WD, I can see I wasn't being very clear; let me try again.the 600 outputs each apply a sigmoid nonlinearity to their input to compute the final output. This forces the output for each of the 600 outputs to be between 0 and 1, but they remain independent (except as linked by the rest of the net). The net learns to predict a CDF or something like it because it's being fed step functions (the encoded labels) as its targets and it try's to match them.
\n\nAs for the custom evaluation function, I'm not sure if that is being used as the objective function – in which case it affects the fit – or just as an evaluation function – in which case it's just reporting on the fit. I actually use Lasagne, because that's what I'm most familiar with, so I'm unclear on some of the details of MXNet.
\n\nPredicting a point estimate using linear regression is definitely more intuitive. However, It's doubtful that it's more powerful, although it's possible it's about the same. It's easy enough to get a point estimate from a CDF if you want one and you also get some estimate of the uncertainty. Whether that estimate of uncertainty will end up being useful in the end is a bit of an open question though. I've tried a couple of times to predict a point estimate and an uncertainty directly using some non-standard objective functions, since that would be cleaner than this approach, but didn't have any luck. Perhaps there is a reason people typically use the standard objective functions ;-).
\n\nThanks for the congrats, I'm doing my best to hang on!
", "rawMarkdown": "\r\n@WD, I can see I wasn't being very clear; let me try again.the 600 outputs each apply a sigmoid nonlinearity to their input to compute the final output. This forces the output for each of the 600 outputs to be between 0 and 1, but they remain independent (except as linked by the rest of the net). The net learns to predict a CDF or something like it because it's being fed step functions (the encoded labels) as its targets and it try's to match them.\r\n\r\nAs for the custom evaluation function, I'm not sure if that is being used as the objective function – in which case it affects the fit – or just as an evaluation function – in which case it's just reporting on the fit. I actually use Lasagne, because that's what I'm most familiar with, so I'm unclear on some of the details of MXNet.\r\n\r\nPredicting a point estimate using linear regression is definitely more intuitive. However, It's doubtful that it's more powerful, although it's possible it's about the same. It's easy enough to get a point estimate from a CDF if you want one and you also get some estimate of the uncertainty. Whether that estimate of uncertainty will end up being useful in the end is a bit of an open question though. I've tried a couple of times to predict a point estimate and an uncertainty directly using some non-standard objective functions, since that would be cleaner than this approach, but didn't have any luck. Perhaps there is a reason people typically use the standard objective functions ;-).\r\n\r\nThanks for the congrats, I'm doing my best to hang on!", "votes": 2 }, { "id": 105619, "postDate": "2016-01-25T09:38:00.800Z", "content": "Hi, the Mxnet is great, I used it get the score 0.022, but it only includes 2D convolution now, when improve it by including 3D ? I really really need 3D convolution to test, thanks !
", "rawMarkdown": "Hi, the Mxnet is great, I used it get the score 0.022, but it only includes 2D convolution now, when improve it by including 3D ? I really really need 3D convolution to test, thanks !", "votes": 2 }, { "id": 105458, "postDate": "2016-01-23T07:24:12.130Z", "content": "@athyssen (and all): did you figure out what the issue was? I have the same problem, i.e. proper learning behavior when running on cpu(0) but no learning progress (and only zeros in the prediction and an error of ~0.8) when I switch to devs = [mx.gpu(0)]. What am I missing?
\n\nThank you!
\n\nyou can add validation data to model fit as below. After that I just grabbed the train and validation errors from the output to generate the plot
\n\ndata_train = mx.io.CSVIter(data_csv="./local_train-64x64-data1.csv", data_shape=(30, 64, 64),\n label_csv="./local_train-stytole.csv", label_shape=(600,),\n batch_size=batch_size)\n\ndata_validate = mx.io.CSVIter(data_csv="./local_test-64x64-data1.csv",data_shape=(30, 64, 64), \n label_csv="./local_test-stytole.csv", label_shape=(600,),\n batch_size=batch_size)\n\nstytole_model.fit(X=data_train, eval_data=data_validate, eval_metric = mx.metric.np(CRPS))\n[quote=WD;104883]
\n\n@EIGSI - many thanks for sharing this. i had overlooked especially the last line in that code. On another topic - how did you create the validation curves that you posted earlier? Did you use the code on https://github.com/dmlc/mxnet/issues/511 (which i am still trying to get to work) or did you take another approach?
\n\n[/quote]
", "rawMarkdown": "you can add validation data to model fit as below. After that I just grabbed the train and validation errors from the output to generate the plot\r\n\r\n\r\n data_train = mx.io.CSVIter(data_csv=\"./local_train-64x64-data1.csv\", data_shape=(30, 64, 64),\r\n label_csv=\"./local_train-stytole.csv\", label_shape=(600,),\r\n batch_size=batch_size)\r\n \r\n data_validate = mx.io.CSVIter(data_csv=\"./local_test-64x64-data1.csv\",data_shape=(30, 64, 64), \r\n label_csv=\"./local_test-stytole.csv\", label_shape=(600,),\r\n batch_size=batch_size)\r\n \r\n stytole_model.fit(X=data_train, eval_data=data_validate, eval_metric = mx.metric.np(CRPS))\r\n\r\n[quote=WD;104883]\r\n\r\n@EIGSI - many thanks for sharing this. i had overlooked especially the last line in that code. On another topic - how did you create the validation curves that you posted earlier? Did you use the code on https://github.com/dmlc/mxnet/issues/511 (which i am still trying to get to work) or did you take another approach? \r\n\r\n[/quote]\r\n", "votes": 2 }, { "id": 104850, "postDate": "2016-01-17T10:54:23.020Z", "content": "I am trying to wrap my head around the Mxnet model. I am struggling to understand the format / shape of the different data files, and would like to test my understanding
\n\nFrom this point my understanding gets hazy. I dont fully understand how we move from the output from the prediction (which has 1048 rows or 1048-30-image-stacks) to the final submission (where we have one row for each of the 200 patients). I would image that we have to do some averaging between different predictions for different stacks for a given patient - but i dont see this in the code. Any help much appreciated
", "rawMarkdown": "I am trying to wrap my head around the Mxnet model. I am struggling to understand the format / shape of the different data files, and would like to test my understanding\r\n\r\n* it seems that there are in total 191645 dcm files in the train directory, across 500 patients\r\n* The script only focuses on these sax directories with a full 30 images. It seems that there are 2641 directories that meet this criterium - leading to 79230 images that we can use \r\n* we create a csv image datafile with one observation for each 30-image-stack, and thus this has the shape of 2641 (rows) and 30*64*64 = 122880 (columns)\r\n* we train this input file against a labelled datafile, the latter being a 2641*600 file. There are 600 columns as we have 600 points on the CPRS distribution that we want to estimate\r\n* we predict using this model on the validation file (validation-64*64) , which has 1048 rows (of 200 patients)\r\n\r\nFrom this point my understanding gets hazy. I dont fully understand how we move from the output from the prediction (which has 1048 rows or 1048-30-image-stacks) to the final submission (where we have one row for each of the 200 patients). I would image that we have to do some averaging between different predictions for different stacks for a given patient - but i dont see this in the code. Any help much appreciated\r\n\r\n", "votes": 2 }, { "id": 104313, "postDate": "2016-01-11T17:11:19.327Z", "content": "Still having no training progress when set to gpu. Are there some other mxnet simple checks to diagnose gpu / cudnn is installed properly?
\n\nwhen using: devs = [mx.gpu()] or [mx.gpu(0)]\nTrain-CRPS stays constant at .8xxxx over all epochs
\n\nwhen using: devs = [ms.cpu()]\nTrain-CRPS decreases as expected.
", "rawMarkdown": "Still having no training progress when set to gpu. Are there some other mxnet simple checks to diagnose gpu / cudnn is installed properly?\r\n\r\nwhen using: devs = [mx.gpu()] or [mx.gpu(0)]\r\nTrain-CRPS stays constant at .8xxxx over all epochs\r\n\r\nwhen using: devs = [ms.cpu()]\r\nTrain-CRPS decreases as expected.", "votes": 2 }, { "id": 104299, "postDate": "2016-01-11T15:41:09.757Z", "content": "Thank you for the tutorial! I installed everything last night and Preprocessing went smoothly. Ran training for the first time this morning and it seems to be executing but with no progress for all epochs. This is running with a GTX970 and I'm not seeing any gpu\\cuda related errors. Any ideas?
\n\nIn [17]: run Train.py
\n\nINFO:root:Start training with [gpu(0)]
\n\nINFO:root:Epoch[0] Train-CRPS=0.875214\nINFO:root:Epoch[0] Time cost=10.447
\n\n.....
\n\nINFO:root:Epoch[64] Train-CRPS=0.875214\nINFO:root:Epoch[64] Time cost=10.378
", "rawMarkdown": "Thank you for the tutorial! I installed everything last night and Preprocessing went smoothly. Ran training for the first time this morning and it seems to be executing but with no progress for all epochs. This is running with a GTX970 and I'm not seeing any gpu\\cuda related errors. Any ideas?\r\n\r\nIn [17]: run Train.py \r\n\r\nINFO:root:Start training with [gpu(0)]\r\n\r\nINFO:root:Epoch[0] Train-CRPS=0.875214\r\nINFO:root:Epoch[0] Time cost=10.447\r\n\r\n.....\r\n\r\nINFO:root:Epoch[64] Train-CRPS=0.875214\r\nINFO:root:Epoch[64] Time cost=10.378\r\n", "votes": 2 }, { "id": 104072, "postDate": "2016-01-08T23:34:14.080Z", "content": "@IAslam, I didn't read that comment as stating that segmented data was required, but rather that the the competition admin (Shannon) felt that the winning solution would likely include segmentation as part of the solution pipeline. Both of the "official" tutorials take this approach; finding the area of the LV in all of the slices and then adding up all of the slices. However, this tutorial, despite taking a more simplistic approach of just training on all of the slices and averaging the results, performs much better than the more sophisticated approaches in tutorials. So, I'm not convinced that the eventual winner will actually segment the slices.
", "rawMarkdown": "@IAslam, I didn't read that comment as stating that segmented data was *required*, but rather that the the competition admin (Shannon) felt that the winning solution would likely include segmentation as part of the solution pipeline. Both of the \"official\" tutorials take this approach; finding the area of the LV in all of the slices and then adding up all of the slices. However, this tutorial, despite taking a more simplistic approach of just training on all of the slices and averaging the results, performs much better than the more sophisticated approaches in tutorials. So, I'm not convinced that the eventual winner will actually segment the slices. ", "votes": 2 }, { "id": 103854, "postDate": "2016-01-07T03:25:56.470Z", "content": "Thanks Franc, and of course thanks Bing for this very interesting tutorial.
\n\nOne suggestion in bold (this allows reprocessing -in case it crashs or to give a try to several preprocessing)\n[code]\n for root, _, files in os.walk(root_path):\n root=root.replace('\\','/')\n files=[s for s in files if ".dcm" in s]\n if len(files) == 0 or not files[0].endswith(".dcm") or root.find("sax") == -1:\n continue\n prefix = files[0].rsplit('-', 1)[0]\n fileset = set(files)\n expected = ["%s-%04d.dcm" % (prefix, i + 1) for i in range(30)]\n if all(x in fileset for x in expected):\n ret.append([root + "/" + x for x in expected])\n # sort for reproduciblity\n return sorted(ret, key = lambda x: x[0])\n[/code]
", "rawMarkdown": "Thanks Franc, and of course thanks Bing for this very interesting tutorial.\r\n\r\nOne suggestion in bold (this allows reprocessing -in case it crashs or to give a try to several preprocessing)\r\n[code]\r\n for root, _, files in os.walk(root_path):\r\n root=root.replace('\\\\','/')\r\n **files=[s for s in files if \".dcm\" in s]**\r\n if len(files) == 0 or not files[0].endswith(\".dcm\") or root.find(\"sax\") == -1:\r\n continue\r\n prefix = files[0].rsplit('-', 1)[0]\r\n fileset = set(files)\r\n expected = [\"%s-%04d.dcm\" % (prefix, i + 1) for i in range(30)]\r\n if all(x in fileset for x in expected):\r\n ret.append([root + \"/\" + x for x in expected])\r\n # sort for reproduciblity\r\n return sorted(ret, key = lambda x: x[0])\r\n[/code]\r\n\r\n", "votes": 2 }, { "id": 103816, "postDate": "2016-01-06T18:47:36.037Z", "content": "Is there any chance that a benevolent spirit would be wiling to setup a AWS Community AMI Instance with MxNet installed? There are some instances available for Mxnet with Julia (N. Virgnian), but it would be great to have e.g. an Ubuntu + Python + MxNet Community AMI....
", "rawMarkdown": "Is there any chance that a benevolent spirit would be wiling to setup a AWS Community AMI Instance with MxNet installed? There are some instances available for Mxnet with Julia (N. Virgnian), but it would be great to have e.g. an Ubuntu + Python + MxNet Community AMI.... ", "votes": 2 }, { "id": 103813, "postDate": "2016-01-06T18:28:48.510Z", "content": "@phunter. many thanks for this. I would love to see a good no-pain tutoral on how to install MxNet on AWS. Might be noob question - but what does star-ed MxNet github mean? also - maybe even more noob - but what is the reason that people prefer MxNet over e.g. Theano/Lasagne or Caffe? thanks!
", "rawMarkdown": "@phunter. many thanks for this. I would love to see a good no-pain tutoral on how to install MxNet on AWS. Might be noob question - but what does star-ed MxNet github mean? also - maybe even more noob - but what is the reason that people prefer MxNet over e.g. Theano/Lasagne or Caffe? thanks! ", "votes": 2 }, { "id": 103532, "postDate": "2016-01-04T02:15:56.277Z", "content": "@EIGSI the last epoch takes some memory so it has the risk of failure since 1GB is kind of small. Some suggestions:
\n\nMXNET_BACKWARD_DO_MIRROR=1 python Train.py
", "rawMarkdown": "@EIGSI the last epoch takes some memory so it has the risk of failure since 1GB is kind of small. Some suggestions:\r\n\r\n0. Try Amazon AWS GPU.\r\n1. change to smaller batch size, 16 or less. Do you want to try 8?\r\n2. MXnet has some magic as I mentioned in https://no2147483647.wordpress.com/2015/12/21/deep-learning-for-hackers-with-mxnet-2/ : the latest MXnet support mirror memory which trades off computing time vs memory usage. If you have the latest MXnet, please use:\r\n\r\nMXNET_BACKWARD_DO_MIRROR=1 python Train.py", "votes": 2 }, { "id": 102439, "postDate": "2015-12-22T17:07:29.550Z", "content": "Yes, Python 2 is less strict than Python3. I recommend to switch to Python 3 as Python 2 is tooooooo old :)
", "rawMarkdown": "Yes, Python 2 is less strict than Python3. I recommend to switch to Python 3 as Python 2 is tooooooo old :)", "votes": 2 }, { "id": 106262, "postDate": "2016-01-29T14:56:53.353Z", "content": "@WD...in your example the network has 0% confidence that the volume is 6,7 or 8 and 10% confident that the volume is less than 1,2,3,4. it is 89% confident that is it is equal to node 5 and 99% confident the volume is less than 6,7,8. The CDF is the integral of the probability distribution function in this case the probability mass function since we are dealing with discrete probabilities. It is the area under the PDF. if you want to get the individual probabilities you will differentiate the CDF and get 0.1, 0, 0, 0, 0.89, 0, 0, 0. cumulatively summing this up left to right gets you the original CDF. These are the probabilities of an 8 node softmax that will produce the CDF in your example. You can see that it is most confident at node 5.
", "rawMarkdown": "@WD...in your example the network has 0% confidence that the volume is 6,7 or 8 and 10% confident that the volume is **less** than 1,2,3,4. it is 89% confident that is it is **equal to** node 5 and 99% confident the volume is **less** than 6,7,8. The CDF is the integral of the probability distribution function in this case the probability mass function since we are dealing with discrete probabilities. It is the area under the PDF. if you want to get the individual probabilities you will differentiate the CDF and get 0.1, 0, 0, 0, 0.89, 0, 0, 0. cumulatively summing this up left to right gets you the original CDF. These are the probabilities of an 8 node softmax that will produce the CDF in your example. You can see that it is most confident at node 5.", "votes": -1 }, { "id": 102712, "postDate": "2015-12-24T21:47:30.247Z", "content": "Thank you for sharing the framework & project. I have managed to ran Preprocessing.py, but Train.py failed with the following: AttributeError: module 'mxnet.io' has no attribute 'CSVIter'
\n\nNote that I am using the pre-built Windows version from here: https://github.com/dmlc/mxnet/releases. This is because it takes a while until my NVidia user gets approved. From what I see on https://github.com/dmlc/mxnet, there is a comment: "[IO] Add CSV Iter" from 3 days ago. This means that this change has not been included yet in the pre-built package, and therefore I cannot use mxnet.io.CSVIter present in the Train.py code.
\n\nIs there any way in which I could get a pre-built package with the latest changes?
\n\nThanks!
", "rawMarkdown": "Thank you for sharing the framework & project. I have managed to ran Preprocessing.py, but Train.py failed with the following: AttributeError: module 'mxnet.io' has no attribute 'CSVIter'\r\n\r\nNote that I am using the pre-built Windows version from here: https://github.com/dmlc/mxnet/releases. This is because it takes a while until my NVidia user gets approved. From what I see on https://github.com/dmlc/mxnet, there is a comment: \"[IO] Add CSV Iter\" from 3 days ago. This means that this change has not been included yet in the pre-built package, and therefore I cannot use mxnet.io.CSVIter present in the Train.py code. \r\n\r\nIs there any way in which I could get a pre-built package with the latest changes?\r\n\r\nThanks!", "votes": -1 }, { "id": 349300, "postDate": "2018-06-28T01:57:15.393Z", "content": "Thx a lot :)
", "rawMarkdown": "Thx a lot :)" }, { "id": 124769, "postDate": "2016-06-22T06:28:46.110Z", "content": "Hi all,
\n\nI would like to modify the crop_resize(img, size) function in Preprocessing.py file.
\n\nFor that I need to get the image path as well. For that, I assume that I need to pass another variable (y) to preproc in function write_data_csv(fname, frames, preproc). So calling the mentioned function would be like:
\n\ntrain_lst = write_data_csv("./train-64x64-data.csv", train_frames, lambda x,y: crop_resize(x, 64,y))
\n\nI am not sure where to add another parameter to preproc.
\n\nSo my question is how to pass each file path along with x ,image matrix, to crop_resize(img, size) function?
\n\nI appreciate any suggestion.
\n\nCheers
\n\nAmin
", "rawMarkdown": "Hi all, \r\n\r\nI would like to modify the **crop_resize(img, size)** function in *Preprocessing.py* file. \r\n\r\nFor that I need to get the image path as well. For that, I assume that I need to pass another variable (y) to **preproc** in function **write_data_csv(fname, frames, preproc)**. So calling the mentioned function would be like:\r\n\r\n**train_lst = write_data_csv(\"./train-64x64-data.csv\", train_frames, lambda x,y: crop_resize(x, 64,y))**\r\n\r\nI am not sure where to add another parameter to **preproc**. \r\n\r\nSo my question is how to pass each file path along with x ,image matrix, to **crop_resize(img, size)** function?\r\n\r\nI appreciate any suggestion.\r\n\r\nCheers\r\n\r\nAmin" }, { "id": 110273, "postDate": "2016-03-04T08:23:54.913Z", "content": "@Maineiac - i was struggling to figure out how to make mx.io.ImageRecordIter work in this context as well as unfortunately
", "rawMarkdown": "@Maineiac - i was struggling to figure out how to make mx.io.ImageRecordIter work in this context as well as unfortunately " }, { "id": 110254, "postDate": "2016-03-04T02:18:38.237Z", "content": "[quote=phunter;110249]
\n\nOne needs to use mx.io.ImageRecordIter (not CSV) for calling these augment online and the IO document page also has explanations http://mxnet.readthedocs.org/en/latest/python/io.html#module-mxnet.io : one can check mxnet.io.ImageRecordIter part and see multiple augment methods which include:
\n\n[/quote]
\n\nThanks, phunter. I read the tutorials and the documentation (even before I posted to the forum :-) )
\n\nThe ImageRecordIter looks like it only handles one image at time, not the situation here where you'd want treat all 30 images from a scan as one larger tensor. That's why I'm asking the question.
\n\nAnyone else have any experience with this sort of situation with mxnet?
", "rawMarkdown": "[quote=phunter;110249]\r\n\r\nOne needs to use mx.io.ImageRecordIter (not CSV) for calling these augment online and the IO document page also has explanations http://mxnet.readthedocs.org/en/latest/python/io.html#module-mxnet.io : one can check mxnet.io.ImageRecordIter part and see multiple augment methods which include:\r\n\r\n[/quote]\r\n\r\nThanks, phunter. I read the tutorials and the documentation (even before I posted to the forum :-) )\r\n\r\nThe ImageRecordIter looks like it only handles one image at time, not the situation here where you'd want treat all 30 images from a scan as one larger tensor. That's why I'm asking the question. \r\n\r\nAnyone else have any experience with this sort of situation with mxnet?\r\n" }, { "id": 110253, "postDate": "2016-03-04T02:07:06.043Z", "content": "Hi guys,
\n\nMaybe a stupid question. How would you continue to train the model without creating a new one with parameters equal to the pre-trained model? I tried something like that:
\n\nmodel = mx.model.FeedForward(ctx=devs,\n symbol = network,\n num_epoch = 5,\n learning_rate = 0.001,\n wd = 0.00001,\n momentum = 0.9\n )\n\nnum_epochs = 10\n\nfor i in range(num_epochs):\n model.fit(X=data_train, eval_data=data_test, eval_metric = mx.metric.np(CRPS))\n pred1 = model.predict(data_local1)\n pred2 = model.predict(data_local2)\n #some maniupulations on pred1 and pred2\nIt appears that in each round of the for loop the model continues to learn, but somehow the predictions pred1 and pred2 do not change from an iteration to iteration. Furthermore, I run into CUDA out of memory problem very fast, which is weird since without the for loop I could be running the model with num_epoch set to whichever number i want.
One needs to use mx.io.ImageRecordIter (not CSV) for calling these augment online and the IO document page also has explanations http://mxnet.readthedocs.org/en/latest/python/io.html#module-mxnet.io : one can check mxnet.io.ImageRecordIter part and see multiple augment methods which include:
\n\n[quote=Maineiac;110243]
\n\n[quote=phunter;110237]
\n\nmx.io.ImageRecordIter support some online augment (rand_crop, rand_mirror etc): http://mxnet.readthedocs.org/en/latest/python/io.html
\n\nand the example github page of MXnet has many ImageRecordIter use cases for reference, e.g. the Imagenet one.
\n\n[/quote]
\n\nThanks for the response.
\n\nFrom what I can tell, there seems to be a difference between those tutorials / examples and what we have in the current situation. The tutorials show images that are at most, 3 dimensions (e.g., 3X64X64, first dimension RGB). In the current case, however, we'd probably want to include all 30 images as an input (so, we'd want something like 30X64X64).
\n\nIn the current mxnet tutorial, this is done by flattening the file into the csv, but then no online augmentation seems possible. In contrast, other deeplearning packages seem to support this sort of representation (e.g., in the Keras tutorial).
\n\nSo, I guess I should have asked my question more specifically: Does anyone know if mxnet can handle online augmentation in this sort of situation where we'd stack the images on top into 30X64X64 tensors?
\n\n[/quote]
", "rawMarkdown": "One needs to use mx.io.ImageRecordIter (not CSV) for calling these augment online and the IO document page also has explanations http://mxnet.readthedocs.org/en/latest/python/io.html#module-mxnet.io : one can check mxnet.io.ImageRecordIter part and see multiple augment methods which include:\r\n\r\n- rand_crop (boolean, optional, default=False) – Augmentation Param: Whether to random crop on the image\r\n- crop_y_start (int, optional, default='-1') – Augmentation Param: Where to nonrandom crop on y.\r\n- crop_x_start (int, optional, default='-1') – Augmentation Param: Where to nonrandom crop on x.\r\n- max_rotate_angle (int, optional, default='0') – Augmentation Param: rotated randomly in [-max_rotate_angle, max_rotate_angle].\r\n- max_aspect_ratio (float, optional, default=0) – Augmentation Param: denotes the max ratio of random aspect ratio augmentation.\r\n- max_shear_ratio (float, optional, default=0) – Augmentation Param: denotes the max random shearing ratio.\r\n- max_crop_size (int, optional, default='-1') – Augmentation Param: Maximum crop size.\r\n- min_crop_size (int, optional, default='-1') – Augmentation Param: Minimum crop size.\r\n- max_random_scale (float, optional, default=1) – Augmentation Param: Maxmum scale ratio.\r\n- min_random_scale (float, optional, default=1) – Augmentation Param: Minimum scale ratio.\r\n- max_img_size (float, optional, default=1e+10) – Augmentation Param: Maxmum image size after resizing.\r\n- min_img_size (float, optional, default=0) – Augmentation Param: Minimum image size after resizing.\r\n- random_h (int, optional, default='0') – Augmentation Param: Maximum value of H channel in HSL color space.\r\n- random_s (int, optional, default='0') – Augmentation Param: Maximum value of S channel in HSL color space.\r\n- random_l (int, optional, default='0') – Augmentation Param: Maximum value of L channel in HSL color space.\r\n- rotate (int, optional, default='-1') – Augmentation Param: Rotate angle.\r\n- fill_value (int, optional, default='255') – Augmentation Param: Maximum value of illumination variation.\r\n- inter_method (int, optional, default='1') – Augmentation Param: 0-NN 1-bilinear 2-cubic 3-area 4-lanczos4 9-auto 10-rand.\r\n- mirror (boolean, optional, default=False) – Augmentation Param: Whether to mirror the image.\r\n- rand_mirror (boolean, optional, default=False) – Augmentation Param: Whether to mirror the image randomly.\r\n- mean_img (string, optional, default='') – Augmentation Param: Mean Image to be subtracted.\r\n- mean_r (float, optional, default=0) – Augmentation Param: Mean value on R channel.\r\n- mean_g (float, optional, default=0) – Augmentation Param: Mean value on G channel.\r\n- mean_b (float, optional, default=0) – Augmentation Param: Mean value on B channel.\r\n- mean_a (float, optional, default=0) – Augmentation Param: Mean value on Alpha channel.\r\n- scale (float, optional, default=1) – Augmentation Param: Scale in color space.\r\n- max_random_contrast (float, optional, default=0) – Augmentation Param: Maximum ratio of contrast variation.\r\n- max_random_illumination (float, optional, default=0) – Augmentation Param: Maximum value of illumination variation.\r\n\r\n[quote=Maineiac;110243]\r\n\r\n[quote=phunter;110237]\r\n\r\nmx.io.ImageRecordIter support some online augment (rand_crop, rand_mirror etc): http://mxnet.readthedocs.org/en/latest/python/io.html \r\n\r\nand the example github page of MXnet has many ImageRecordIter use cases for reference, e.g. the Imagenet one.\r\n\r\n[/quote]\r\n\r\nThanks for the response.\r\n\r\nFrom what I can tell, there seems to be a difference between those tutorials / examples and what we have in the current situation. The tutorials show images that are at most, 3 dimensions (e.g., 3X64X64, first dimension RGB). In the current case, however, we'd probably want to include all 30 images as an input (so, we'd want something like 30X64X64). \r\n\r\nIn the current mxnet tutorial, this is done by flattening the file into the csv, but then no online augmentation seems possible. In contrast, other deeplearning packages seem to support this sort of representation (e.g., in the Keras tutorial).\r\n\r\nSo, I guess I should have asked my question more specifically: Does anyone know if mxnet can handle online augmentation in this sort of situation where we'd stack the images on top into 30X64X64 tensors?\r\n\r\n\r\n\r\n[/quote]" }, { "id": 110243, "postDate": "2016-03-04T01:05:13.450Z", "content": "[quote=phunter;110237]
\n\nmx.io.ImageRecordIter support some online augment (rand_crop, rand_mirror etc): http://mxnet.readthedocs.org/en/latest/python/io.html
\n\nand the example github page of MXnet has many ImageRecordIter use cases for reference, e.g. the Imagenet one.
\n\n[/quote]
\n\nThanks for the response.
\n\nFrom what I can tell, there seems to be a difference between those tutorials / examples and what we have in the current situation. The tutorials show images that are at most, 3 dimensions (e.g., 3X64X64, first dimension RGB). In the current case, however, we'd probably want to include all 30 images as an input (so, we'd want something like 30X64X64).
\n\nIn the current mxnet tutorial, this is done by flattening the file into the csv, but then no online augmentation seems possible. In contrast, other deeplearning packages seem to support this sort of representation (e.g., in the Keras tutorial).
\n\nSo, I guess I should have asked my question more specifically: Does anyone know if mxnet can handle online augmentation in this sort of situation where we'd stack the images on top into 30X64X64 tensors?
", "rawMarkdown": "[quote=phunter;110237]\r\n\r\nmx.io.ImageRecordIter support some online augment (rand_crop, rand_mirror etc): http://mxnet.readthedocs.org/en/latest/python/io.html \r\n\r\nand the example github page of MXnet has many ImageRecordIter use cases for reference, e.g. the Imagenet one.\r\n\r\n[/quote]\r\n\r\nThanks for the response.\r\n\r\nFrom what I can tell, there seems to be a difference between those tutorials / examples and what we have in the current situation. The tutorials show images that are at most, 3 dimensions (e.g., 3X64X64, first dimension RGB). In the current case, however, we'd probably want to include all 30 images as an input (so, we'd want something like 30X64X64). \r\n\r\nIn the current mxnet tutorial, this is done by flattening the file into the csv, but then no online augmentation seems possible. In contrast, other deeplearning packages seem to support this sort of representation (e.g., in the Keras tutorial).\r\n\r\nSo, I guess I should have asked my question more specifically: Does anyone know if mxnet can handle online augmentation in this sort of situation where we'd stack the images on top into 30X64X64 tensors?\r\n\r\n" }, { "id": 110237, "postDate": "2016-03-04T00:24:29.483Z", "content": "[quote=Maineiac;110111]
\n\nHi all,
\n\nI think I already know that the answer to this question is "No," but has anyone tried online image augmentation with this tutorial?
\n\nHas anyone been using mx.io.ImageRecordIter?
\n\n[/quote]\nmx.io.ImageRecordIter support some online augment (rand_crop, rand_mirror etc): http://mxnet.readthedocs.org/en/latest/python/io.html
\n\nand the example github page of MXnet has many ImageRecordIter use cases for reference, e.g. the Imagenet one.
", "rawMarkdown": "[quote=Maineiac;110111]\r\n\r\nHi all,\r\n\r\nI think I already know that the answer to this question is \"No,\" but has anyone tried online image augmentation with this tutorial?\r\n\r\nHas anyone been using mx.io.ImageRecordIter? \r\n\r\n[/quote]\r\nmx.io.ImageRecordIter support some online augment (rand_crop, rand_mirror etc): http://mxnet.readthedocs.org/en/latest/python/io.html \r\n\r\nand the example github page of MXnet has many ImageRecordIter use cases for reference, e.g. the Imagenet one." }, { "id": 110111, "postDate": "2016-03-03T01:50:55.807Z", "content": "Hi all,
\n\nI think I already know that the answer to this question is "No," but has anyone tried online image augmentation with this tutorial?
\n\nHas anyone been using mx.io.ImageRecordIter?
", "rawMarkdown": "Hi all,\r\n\r\nI think I already know that the answer to this question is \"No,\" but has anyone tried online image augmentation with this tutorial?\r\n\r\nHas anyone been using mx.io.ImageRecordIter? " }, { "id": 110026, "postDate": "2016-03-02T07:07:34.693Z", "content": "Acquire Nvidia.
", "rawMarkdown": "Acquire Nvidia." }, { "id": 110022, "postDate": "2016-03-02T05:03:24.373Z", "content": "Thanks for a great tutorial!
\n\nI would like to try the in memory setting for accelerate the learning process but I'm stuck.\nWould be great if someone can help me out??
", "rawMarkdown": "Thanks for a great tutorial!\r\n\r\nI would like to try the in memory setting for accelerate the learning process but I'm stuck.\r\nWould be great if someone can help me out??" }, { "id": 109385, "postDate": "2016-02-25T16:14:06.077Z", "content": "@Tim Hochberg
\n\nThanks for the speedy response! What you suggest is what I am currently doing, so I guess I'll just stick with that.
", "rawMarkdown": "@Tim Hochberg\r\n\r\nThanks for the speedy response! What you suggest is what I am currently doing, so I guess I'll just stick with that. " }, { "id": 109365, "postDate": "2016-02-25T14:14:50.463Z", "content": "I hope that everyone is enjoying the competition! I'm trying to implement something similar to this in Theano and I'm having difficulties with the loss function. It doesn't like the range() here,
\n\ndef CRPS(label, pred):\n """ Custom evaluation metric on CRPS.\n """\n for i in range(pred.shape[0]):\n for j in range(pred.shape[1] - 1):\n if pred[i, j] > pred[i, j + 1]:\n pred[i, j + 1] = pred[i, j]\n return np.sum(np.square(label - pred)) / label.size\nIf I understand correctly, pred.shape[0] would be the batch size and pred.shape[1] would be 600 for the number of columns?
", "rawMarkdown": "I hope that everyone is enjoying the competition! I'm trying to implement something similar to this in Theano and I'm having difficulties with the loss function. It doesn't like the range() here, \r\n\r\n def CRPS(label, pred):\r\n \"\"\" Custom evaluation metric on CRPS.\r\n \"\"\"\r\n for i in range(pred.shape[0]):\r\n for j in range(pred.shape[1] - 1):\r\n if pred[i, j] > pred[i, j + 1]:\r\n pred[i, j + 1] = pred[i, j]\r\n return np.sum(np.square(label - pred)) / label.size\r\n\r\n\r\nIf I understand correctly, pred.shape[0] would be the batch size and pred.shape[1] would be 600 for the number of columns?" }, { "id": 109332, "postDate": "2016-02-25T05:15:59.357Z", "content": "[quote=liubenyuan;109323]
\n\n[quote=LoneStar;109310]
\n\nHi there, I have pip installed the pydicom package, but still get ImportError: No module named dicom.\nAny idea what's going wrong.
\n\nTraceback (most recent call last):\n File "example/kaggle-ndsb2/Preprocessing.py", line 11, in \n import dicom\nImportError: No module named dicom
\n\n[/quote]
\n\nyou might modify Preprocessing.py, line 11, import pydicom as dicom
\n\n[/quote]\nThank you. I've tried taht, but it will say: ImportError: No module named pydicom
", "rawMarkdown": "[quote=liubenyuan;109323]\r\n\r\n[quote=LoneStar;109310]\r\n\r\nHi there, I have pip installed the pydicom package, but still get ImportError: No module named dicom.\r\nAny idea what's going wrong.\r\n\r\nTraceback (most recent call last):\r\n File \"example/kaggle-ndsb2/Preprocessing.py\", line 11, in[quote=phunter;109312]
\n\n@LoneStar which system do you use? OSX or ubuntu?
\n\n[/quote]\nI am using ubuntu.
", "rawMarkdown": "[quote=phunter;109312]\r\n\r\n@LoneStar which system do you use? OSX or ubuntu? \r\n\r\n[/quote]\r\nI am using ubuntu." }, { "id": 109323, "postDate": "2016-02-25T03:28:23.897Z", "content": "[quote=LoneStar;109310]
\n\nHi there, I have pip installed the pydicom package, but still get ImportError: No module named dicom.\nAny idea what's going wrong.
\n\nTraceback (most recent call last):\n File "example/kaggle-ndsb2/Preprocessing.py", line 11, in \n import dicom\nImportError: No module named dicom
\n\n[/quote]
\n\nyou might modify Preprocessing.py, line 11, import pydicom as dicom
", "rawMarkdown": "[quote=LoneStar;109310]\r\n\r\nHi there, I have pip installed the pydicom package, but still get ImportError: No module named dicom.\r\nAny idea what's going wrong.\r\n\r\nTraceback (most recent call last):\r\n File \"example/kaggle-ndsb2/Preprocessing.py\", line 11, in@cocoding you might want to install MXnet on a GPU machine, either physical or AWS. AWS tutorial: https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ Physical machine: https://no2147483647.wordpress.com/2015/12/07/deep-learning-for-hackers-with-mxnet-1/
", "rawMarkdown": "@cocoding you might want to install MXnet on a GPU machine, either physical or AWS. AWS tutorial: https://no2147483647.wordpress.com/2016/01/16/setup-amazon-aws-gpu-instance-with-mxnet/ Physical machine: https://no2147483647.wordpress.com/2015/12/07/deep-learning-for-hackers-with-mxnet-1/" }, { "id": 109312, "postDate": "2016-02-25T01:48:03.200Z", "content": "@LoneStar which system do you use? OSX or ubuntu?
", "rawMarkdown": "@LoneStar which system do you use? OSX or ubuntu? " }, { "id": 109310, "postDate": "2016-02-25T01:30:18.867Z", "content": "Hi there, I have pip installed the pydicom package, but still get ImportError: No module named dicom.\nAny idea what's going wrong.
\n\nTraceback (most recent call last):\n File "example/kaggle-ndsb2/Preprocessing.py", line 11, in \n import dicom\nImportError: No module named dicom
", "rawMarkdown": "Hi there, I have pip installed the pydicom package, but still get ImportError: No module named dicom.\r\nAny idea what's going wrong.\r\n\r\nTraceback (most recent call last):\r\n File \"example/kaggle-ndsb2/Preprocessing.py\", line 11, in@athyssen, I have exactly the same issue. Please, let me know if you found a solution?
\n\n[quote=athyssen;104313]
\n\nStill having no training progress when set to gpu. Are there some other mxnet simple checks to diagnose gpu / cudnn is installed properly?
\n\nwhen using: devs = [mx.gpu()] or [mx.gpu(0)]\nTrain-CRPS stays constant at .8xxxx over all epochs
\n\nwhen using: devs = [ms.cpu()]\nTrain-CRPS decreases as expected.
\n\n[/quote]
", "rawMarkdown": "@athyssen, I have exactly the same issue. Please, let me know if you found a solution?\r\n\r\n[quote=athyssen;104313]\r\n\r\nStill having no training progress when set to gpu. Are there some other mxnet simple checks to diagnose gpu / cudnn is installed properly?\r\n\r\nwhen using: devs = [mx.gpu()] or [mx.gpu(0)]\r\nTrain-CRPS stays constant at .8xxxx over all epochs\r\n\r\nwhen using: devs = [ms.cpu()]\r\nTrain-CRPS decreases as expected.\r\n\r\n[/quote]\r\n" }, { "id": 109084, "postDate": "2016-02-23T05:25:53.830Z", "content": "What is the preparatory work needed to try this tutorial?
\n\nI have downloaded the data and installed python. What other tools should I get ready?
\n\nDeep learning, GPU, object segmentation, etc, I am totally new to these things. Appreciate any help.
\n\n[quote=Bing Xu;102412]
\n\nHi All,
\n\nWe just make an end-to-end deep learning tutorial with lb score 0.0392:
\n\nhttps://github.com/dmlc/mxnet/tree/master/example/kaggle-ndsb2
\n\nNotice this is a very simple model with no attempt to optimize the structure or hyper parameters, you can build fantastic network based on it. While this tutorial is written in python, MXNet comes with support for other popular languages such as R and Julia which can also be used. You are more than welcomed to try and contribute back to this example.
\n\nBests,
\n\nBing
\n\n[/quote]
", "rawMarkdown": "What is the preparatory work needed to try this tutorial?\r\n\r\nI have downloaded the data and installed python. What other tools should I get ready? \r\n\r\nDeep learning, GPU, object segmentation, etc, I am totally new to these things. Appreciate any help.\r\n\r\n[quote=Bing Xu;102412]\r\n\r\nHi All,\r\n\r\nWe just make an end-to-end deep learning tutorial with lb score 0.0392:\r\n\r\nhttps://github.com/dmlc/mxnet/tree/master/example/kaggle-ndsb2\r\n\r\n\r\nNotice this is a very simple model with no attempt to optimize the structure or hyper parameters, you can build fantastic network based on it. While this tutorial is written in python, MXNet comes with support for other popular languages such as R and Julia which can also be used. You are more than welcomed to try and contribute back to this example.\r\n\r\nBests,\r\n\r\nBing\r\n\r\n[/quote]\r\n" }, { "id": 108892, "postDate": "2016-02-21T05:55:28.593Z", "content": "[quote=Kosiński KM;108872]
\n\nDoes anyone of you know how to impose regularization condition on weights using the mxnet abstraction? It seems to me that this package does not implement such a functionality.
[/quote]
\n\nAll the optimizers (sgd, adam) in mxnet have a wd parameter, that is the coefficient in front of L2 regularization term. If omitted, no regularization is applied to the weights.
Does anyone of you know how to impose regularization condition on weights using the mxnet abstraction? It seems to me that this package does not implement such a functionality.
Traceback (most recent call last):\n File "Train.py", line 159, in <module>\n systole_result = accumulate_result("../data/validate-label.csv", systole_prob)\n File "Train.py", line 145, in accumulate_result\n line = fi.__next__() # Python2: line = fi.next()\nStopIteration\nI saw that someone tried to mention this error early on. I am late in the game and just testing this code.\nI can see the file and its content and I am not sure why I am running in to this exception.
\n\nI tried both python2 and python3. Same issue. Any one would like to give insight?
", "rawMarkdown": " Traceback (most recent call last):\r\n File \"Train.py\", line 159, ini am running linear regression with the model as per below - but still have the weird problem that the error in my predictions are not centered around zero. The mean systole is on average -10 to low, and the mean diastole on average -30 to low. I do not understand why the network does not adjust for this in the bias term. See snapshot below. any help or advice much appreciated!
\n\ndef get_lenet():\n""" A lenet style net, takes difference of each frame as input.\n"""\nsource = mx.sym.Variable("data")\nsource = (source - 128) * (1.0/128)\nframes = mx.sym.SliceChannel(source, num_outputs=30)\ndiffs = [frames[i+1] - frames[i] for i in range(29)] \nsource = mx.sym.Concat(*diffs)\nnet = mx.sym.Convolution(source, kernel=(3, 3), pad=(1,1), stride=(1,1), num_filter=40)\nnet = mx.sym.BatchNorm(net, fix_gamma=True)\nnet = mx.sym.Activation(net, act_type="relu")\nnet = mx.sym.Dropout(net, p=0.25) \nnet = mx.sym.Pooling(net, pool_type="max", kernel=(2,2), stride=(2,2))\nnet = mx.sym.Convolution(net, kernel=(3, 3), pad=(1,1), stride=(1,1), num_filter=40)\nnet = mx.sym.BatchNorm(net, fix_gamma=True)\nnet = mx.sym.Activation(net, act_type="relu")\nnet = mx.sym.Dropout(net, p=0.25) \nnet = mx.sym.Pooling(net, pool_type="max", kernel=(2,2), stride=(2,2))\nflatten = mx.symbol.Flatten(net)\nflatten = mx.sym.Activation(flatten, act_type="relu") \nflatten = mx.symbol.Dropout(flatten)\nfc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=1024)\nfc1 = mx.sym.Activation(fc1, act_type="relu")\nfc1 = mx.symbol.Dropout(fc1)\nfc2 = mx.symbol.FullyConnected(data=fc1, num_hidden=1)\nreturn mx.symbol.LinearRegressionOutput(data=fc2,name='softmax')\n\nlearning_rate = 0.0000001,\nwd = 0.00001,\n@Franc. thanks so much. should one add as well a convolutional relu layer between the flatten and the fc1? as per below? or is that not correct / needed?
\n\n flatten = mx.symbol.Flatten(net)\n flatten = mx.symbol.Dropout(flatten)\n flatten = mx.sym.Activation(flatten, act_type="relu") ###to be checked\n fc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=1024)\n fc1 = mx.sym.Activation(fc1, act_type="relu")\n fc1 = mx.symbol.Dropout(fc1)\n fc2 = mx.symbol.FullyConnected(data=fc1, num_hidden=1)\n[quote=WD;106519]
\n\nLinear regression. Has anyone moved the code to a linear regression setup? I have tried to do so - but get predictions that are structurally a little off. I have basically adapted the model as follows (and changed hte shape of the label data). Would love advice / suggestions on whether this is the right approach:
\n\nflatten = mx.symbol.Dropout(flatten)\nfc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=1024) \nfc2 = mx.symbol.FullyConnected(data=fc1, num_hidden=1)\nreturn mx.symbol.LinearRegressionOutput(data=fc2,name='softmax')\n[/quote]
\n\nIn the code above, you actually do not have any nonlinearity between fc1 and fc2. Put an activation and a dropout layer between fc1 and fc2; for example:
\n\nflatten = mx.symbol.Dropout(flatten)\nfc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=1024) \n#-------\nfc1 = mx.sym.Activation(fc1, act_type="relu")\nfc1 = mx.symbol.Dropout(fc1)\n#------\nfc2 = mx.symbol.FullyConnected(data=fc1, num_hidden=1)\nreturn mx.symbol.LinearRegressionOutput(data=fc2,name='softmax')\nLinear regression. Has anyone moved the code to a linear regression setup? I have tried to do so - but get predictions that are structurally a little off. I have basically adapted the model as follows (and changed hte shape of the label data). Would love advice / suggestions on whether this is the right approach:
\n\nflatten = mx.symbol.Dropout(flatten)\nfc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=1024) \nfc2 = mx.symbol.FullyConnected(data=fc1, num_hidden=1)\nreturn mx.symbol.LinearRegressionOutput(data=fc2,name='softmax')\nWhile reading sources of Train.py I have noticed that mxnet learns on labels, which are CDF, but submission is generated as if labels, which are computed on validation data, would be PDF instead. Is it so, or have I missed anything? Now I'm wondering, how this tutorial can produce any meaningful results.
\n\nRelevant lines are below. Note that systole_encode is constructed as CDF while training, but probabilities are used and accumulated while testing.
Training:
\n\ndef encode_label(label_data):\n systole = label_data[:, 1]\n ...\n systole_encode = np.array([\n (x < np.arange(600)) for x in systole\n ], dtype=np.uint8)\n ...\n\ndef encode_csv(label_csv, systole_csv, diastole_csv):\n systole_encode, diastole_encode = encode_label(np.loadtxt(label_csv, delimiter=","))\n ...\n\nencode_csv("./train-label.csv", "./train-systole.csv", "./train-diastole.csv")\n...\ndata_train = mx.io.CSVIter(data_csv="./train-64x64-data.csv", data_shape=(30, 64, 64),\n label_csv="./train-systole.csv", label_shape=(600,),\n batch_size=batch_size)\n...\nsystole_model.fit(X=data_train, eval_metric = mx.metric.np(CRPS))\nTesting:
\n\nsystole_prob = systole_model.predict(data_validate)\n...\nsystole_result = accumulate_result("./validate-label.csv", systole_prob)\n...\nfor line in fi:\n idx = line[0]\n key, target = idx.split('_')\n ...\n out.extend(list(submission_helper(systole_result[key])))\n ...\n fo.writerow(out)\nHi guys,
\n\nI have a question pertaining to an error previously reported by @Matthew Tubs, @Scott Smith and @Jon Carlisle. There hasn't been any definite solution given here, so I wanted to reiterate this problem, since I haven't found one yet as well.
\n\nBasically the pre-processing script finishes with the following error (when run on Windows, Python 3.5 64-bit)
\n\nIndexError: list index out of range,
which in turn is a result of invoking the very last line of the script, i.e.,:
\n\nsplit_csv("..\\\\train-64x64-data.csv", split_to_train, "..\\\\local_train-64x64-data.csv", "..\\\\local_test-64x64-data.csv")
The root cause of the problem (as noted in prrevious posts) is that:
\n\n[quote=Jon Carlisle;104904]
\n\nI'm on the last line of code and reviewing #2 in this post to try and solve my index issue. For me, he train_frames has a length of 5,293 yet train-64x64-data.csv has double that which I believe is why there is an index error. Am I understanding this correctly...how many rows should I have in train-64x64-data.csv?
\n\n[/quote]
\n\nIndeed the file train-64x64-data.csv is of length 2*5,293 and its rows basically look as follows. First there is a row corresponding to a jpg file that looks more or less something like that:
\n\n'2,2,2,2,2,2,74,92,87,79,59,26,10,1,.... lots of numbers,....,4,5,8,7,3,6,2,9,5,6,3,2\\n'
and then the next line is empty and looks as follows:
\n\n'\\n'
The same structure is in the resulting validate-64x64-data.csv. I believe that the blank lines with '\\n' are in both files by a mistake.
Would you happen to know, which line of the write_data_csv function introduces that error?
That is a great question! I was wondering the same. It occurred to me that I need to do some manual pre-processing to get a real score improvement but there are just too many images and little time!
\n\nCongrats for 22nd place, that is much better than mine, any chance we could hear about some of your secrets?
\n\n[quote=Bartek;106075]
\n\nI have one question about MxNet and Keras approach. Why it works?:)
\n\nWhy it should not work? (as this approach is different that obvious one, presented here https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/18516/systole-and-diastole-volume-calculation-from-series-of-2d-segmented-image/105404#post105404)
\n\nWhy it works:
\n\nDo you agree with me or I do not understand the competition idea?:)
\n\n[/quote]
", "rawMarkdown": "That is a great question! I was wondering the same. It occurred to me that I need to do some manual pre-processing to get a real score improvement but there are just too many images and little time!\r\n\r\nCongrats for 22nd place, that is much better than mine, any chance we could hear about some of your secrets?\r\n\r\n[quote=Bartek;106075]\r\n\r\nI have one question about MxNet and Keras approach. Why it works?:)\r\n\r\n\r\nWhy it should not work? (as this approach is different that obvious one, presented here https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/18516/systole-and-diastole-volume-calculation-from-series-of-2d-segmented-image/105404#post105404)\r\n\r\n- the input to the network is a time frames. So it show how the heart is moving. But it does not show the volume of heart\r\n- assigning the volume for each \"sax\" folder is strange too. I can not understand that idea:)\r\n\r\nWhy it works:\r\n\r\n- I think that it does not care about the heart at all:) I think that CNN found some features from chest and try to estimate volumes based on that features. And additionally the heart is really small at 64x64 images. I can not explain it in other way.\r\n\r\nDo you agree with me or I do not understand the competition idea?:)\r\n\r\n[/quote]\r\n" }, { "id": 106086, "postDate": "2016-01-28T11:48:59.353Z", "content": "Thanks for the info @Scott!
\n\n[quote=Scott Smith;105981]
\n\n@Kosinski:\nH/W: Intel i7 5500U CPU, nVIDIA GeForce 840M GPU, 250 gb SSD & 8 gb RAM.\nS/W: Ubuntu 15.04 / Python 3.5x / CUDA 7.5 / CuDNN / openBLAS / mxnet / Anaconda / ...
\n\nFor the tutorial, I think preprocessing took about 1 hour. Training on the CPU took ~26 hours. Training on the GPU took ~2 hours.
\n\nScott
\n\n[/quote]
", "rawMarkdown": "Thanks for the info @Scott!\r\n\r\n[quote=Scott Smith;105981]\r\n\r\n@Kosinski:\r\nH/W: Intel i7 5500U CPU, nVIDIA GeForce 840M GPU, 250 gb SSD & 8 gb RAM.\r\nS/W: Ubuntu 15.04 / Python 3.5x / CUDA 7.5 / CuDNN / openBLAS / mxnet / Anaconda / ...\r\n\r\nFor the tutorial, I think preprocessing took about 1 hour. Training on the CPU took ~26 hours. Training on the GPU took ~2 hours.\r\n\r\nScott\r\n\r\n[/quote]\r\n" }, { "id": 106075, "postDate": "2016-01-28T10:22:58.703Z", "content": "I have one question about MxNet and Keras approach. Why it works?:)
\n\nWhy it should not work? (as this approach is different that obvious one, presented here https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/18516/systole-and-diastole-volume-calculation-from-series-of-2d-segmented-image/105404#post105404)
\n\nWhy it works:
\n\nDo you agree with me or I do not understand the competition idea?:)
", "rawMarkdown": "I have one question about MxNet and Keras approach. Why it works?:)\r\n\r\n\r\nWhy it should not work? (as this approach is different that obvious one, presented here https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/18516/systole-and-diastole-volume-calculation-from-series-of-2d-segmented-image/105404#post105404)\r\n\r\n- the input to the network is a time frames. So it show how the heart is moving. But it does not show the volume of heart\r\n- assigning the volume for each \"sax\" folder is strange too. I can not understand that idea:)\r\n\r\nWhy it works:\r\n\r\n- I think that it does not care about the heart at all:) I think that CNN found some features from chest and try to estimate volumes based on that features. And additionally the heart is really small at 64x64 images. I can not explain it in other way.\r\n\r\nDo you agree with me or I do not understand the competition idea?:)" }, { "id": 106038, "postDate": "2016-01-28T01:09:34.100Z", "content": "@Tim, thanks, I am not using nolearn just plain old lasagne. I have a function to iterate batches I'll try to put the augmentation code there. I actually thought about it but I was just uncomfortable manipulating the nicely packaged stacks of image tensors and not individual images one can manipulate with opencv.
", "rawMarkdown": "@Tim, thanks, I am not using nolearn just plain old lasagne. I have a function to iterate batches I'll try to put the augmentation code there. I actually thought about it but I was just uncomfortable manipulating the nicely packaged stacks of image tensors and not individual images one can manipulate with opencv. " }, { "id": 106026, "postDate": "2016-01-27T23:20:26.563Z", "content": "@DavidGbodiOdaibo, are you using nolearn? If you are using nolearn and you aren't already using batch iterators, then take a look at those. That's a way to do on the fly augmentation. I can point you to some examples, but we should probably start a new topic rather than hijacking this thread.
", "rawMarkdown": "@DavidGbodiOdaibo, are you using nolearn? If you are using nolearn and you aren't already using batch iterators, then take a look at those. That's a way to do on the fly augmentation. I can point you to some examples, but we should probably start a new topic rather than hijacking this thread." }, { "id": 106000, "postDate": "2016-01-27T21:17:52.527Z", "content": "I don't know why I am giving away all my secrets nobody gives me any secrets. Another trick is if you average all the softmax predictions to generate a single CDF. If you are a gambling man and you want to know the most probable volume and convert the CDF into a single step function. You can simply compute the derivatives of the CDF and pick the point that has the highest gradient. That will be point where the slope of the function is greatest and the collective averages are most confident. Set all values in your file to 1 after this point.
", "rawMarkdown": "I don't know why I am giving away all my secrets nobody gives me any secrets. Another trick is if you average all the softmax predictions to generate a single CDF. If you are a gambling man and you want to know the most probable volume and convert the CDF into a single step function. You can simply compute the derivatives of the CDF and pick the point that has the highest gradient. That will be point where the slope of the function is greatest and the collective averages are most confident. Set all values in your file to 1 after this point." }, { "id": 105991, "postDate": "2016-01-27T20:37:24.633Z", "content": "@WD ...as a side comment... if you highlight both Systole and Diastole records in your submission file in excel, you can easily visualize the cumulative distribution function by inserting a line chart. The data you posted looks like a record from the submission file. The CDF is summing the softmax probabilities left to right from 0 -600, not sure if they are using softmax in this tutorial.
", "rawMarkdown": "@WD ...as a side comment... if you highlight both Systole and Diastole records in your submission file in excel, you can easily visualize the cumulative distribution function by inserting a line chart. The data you posted looks like a record from the submission file. The CDF is summing the softmax probabilities left to right from 0 -600, not sure if they are using softmax in this tutorial." }, { "id": 105985, "postDate": "2016-01-27T20:07:08.520Z", "content": "I am using Lasagne too, and I can't get my model's training loss to the absolute minimum I know it is capable of because I am augmenting the data outside training and my dataset is too big to fit in GPUs memory, so I am alternate training on different partitions of the data, this makes the network very confused. Once i figure out how to augment the data in memory. @Tim watch your back :)
", "rawMarkdown": "I am using Lasagne too, and I can't get my model's training loss to the absolute minimum I know it is capable of because I am augmenting the data outside training and my dataset is too big to fit in GPUs memory, so I am alternate training on different partitions of the data, this makes the network very confused. Once i figure out how to augment the data in memory. @Tim watch your back :)" }, { "id": 105982, "postDate": "2016-01-27T19:54:05.493Z", "content": "@Tim. what's the secret? I promise not to tell anyone :)
", "rawMarkdown": "@Tim. what's the secret? I promise not to tell anyone :)" }, { "id": 105981, "postDate": "2016-01-27T19:50:42.857Z", "content": "[quote=Kosiński KM;105942]\nHow long (in terms of computing time) does it typically take you to go through the pre-processing step of this tutorial? I left the script running over night, but it wasn't finished when I woke up. I expected it to run for quite some time, but not that long to be honest...\n[/quote]
\n\n@Kosinski:\nH/W: Intel i7 5500U CPU, nVIDIA GeForce 840M GPU, 250 gb SSD & 8 gb RAM.\nS/W: Ubuntu 15.04 / Python 3.5x / CUDA 7.5 / CuDNN / openBLAS / mxnet / Anaconda / ...
\n\nFor the tutorial, I think preprocessing took about 1 hour. Training on the CPU took ~26 hours. Training on the GPU took ~2 hours.
\n\nScott
", "rawMarkdown": "[quote=Kosiński KM;105942]\r\nHow long (in terms of computing time) does it typically take you to go through the pre-processing step of this tutorial? I left the script running over night, but it wasn't finished when I woke up. I expected it to run for quite some time, but not that long to be honest...\r\n[/quote]\r\n\r\n@Kosinski:\r\nH/W: Intel i7 5500U CPU, nVIDIA GeForce 840M GPU, 250 gb SSD & 8 gb RAM.\r\nS/W: Ubuntu 15.04 / Python 3.5x / CUDA 7.5 / CuDNN / openBLAS / mxnet / Anaconda / ...\r\n\r\nFor the tutorial, I think preprocessing took about 1 hour. Training on the CPU took ~26 hours. Training on the GPU took ~2 hours.\r\n\r\nScott" }, { "id": 105978, "postDate": "2016-01-27T19:32:13.920Z", "content": "@Tim. thanks so much - much appreciated. hopefully will be able to return favor soon. W
", "rawMarkdown": "@Tim. thanks so much - much appreciated. hopefully will be able to return favor soon. W " }, { "id": 105968, "postDate": "2016-01-27T18:52:40.903Z", "content": "@Tim. That helps. When you say sigmoid, do you mean the cumulative distribution function? Does the algorithm "know" that we are looking for a cumulative distribution functoin cause we have set the custom evaluation criterium to CRPS? Or is this the standard output of the log.regression.softmax setting? (i guess the former, but not 100% sure). Wondering out loud whether the Keras tutoral approach of using a linear regression to determine a point estimate and subsequent translation into a PDF / CDF is potentially slightly more intuitive and powerful, but that is seperate topic... let me know your thoughts! (congrats with the #1 spot as well)
", "rawMarkdown": "@Tim. That helps. When you say sigmoid, do you mean the cumulative distribution function? Does the algorithm \"know\" that we are looking for a cumulative distribution functoin cause we have set the custom evaluation criterium to CRPS? Or is this the standard output of the log.regression.softmax setting? (i guess the former, but not 100% sure). Wondering out loud whether the Keras tutoral approach of using a linear regression to determine a point estimate and subsequent translation into a PDF / CDF is potentially slightly more intuitive and powerful, but that is seperate topic... let me know your thoughts! (congrats with the #1 spot as well)" }, { "id": 105963, "postDate": "2016-01-27T18:27:03.060Z", "content": "Softmax output. This might be elementary, but still i am having trouble figuring this out. The Mxnet model creates a mx.symbol.LogisticRegressionOutput with a softmax function. I had previously interpreted the output as a probability distribution over the 600 possible outcomes, normalized such that it sums to 1.
\n\nHowever, i might be wrong in my understanding, as the diastole_prob object (the prediction of the model on the validation set) upon inspection of the first row / first observations looks like per below. This row of 600 observations neither is ascending (like a CDF) nor sums to 1. So i am bit lost - can anyone help me and share their perspective? Much appreciated. W
\n\nIn [16]: diastole_prob[1,:]\nOut[16]:\narray([ 0.05217434, 0.0474867 , 0.08093406, 0.05017258, 0.06756466,\n 0.07113108, 0.05732923, 0.0514702 , 0.07549778, 0.09578432,\n 0.06039102, 0.06391794, 0.07802723, 0.06751554, 0.05773542,\n 0.0845582 , 0.05266474, 0.06205872, 0.07586135, 0.05797536,\n 0.09539511, 0.07857879, 0.0782773 , 0.07885416, 0.07488552,\n 0.05688406, 0.04375266, 0.08723985, 0.09554416, 0.13030726,\n 0.08048934, 0.07286929, 0.06500199, 0.08452114, 0.05599517,\n 0.06534485, 0.12171374, 0.07284499, 0.07555397, 0.05720131,\n 0.05926706, 0.08990043, 0.05770419, 0.06937158, 0.10790928,\n 0.07708788, 0.13347004, 0.07827558, 0.06507879, 0.07119309,\n 0.06509305, 0.07390443, 0.11008534, 0.10490668, 0.08557517,\n 0.08783437, 0.07303309, 0.08287172, 0.08533155, 0.10156922,\n 0.09635049, 0.08800728, 0.0730741 , 0.08773415, 0.1098076 ,\n 0.11432482, 0.07831605, 0.06428145, 0.08088743, 0.09629309,\n 0.08714074, 0.04377145, 0.07941946, 0.12478558, 0.10927155,\n 0.13440026, 0.14776656, 0.05986561, 0.07887186, 0.10276607,\n 0.07350686, 0.13926943, 0.08472204, 0.10816599, 0.08936805,\n 0.0781935 , 0.09617887, 0.1250304 , 0.10030739, 0.11530763,\n 0.12390044, 0.10760392, 0.1366507 , 0.09015703, 0.14969452,\n 0.13367297, 0.17487478, 0.15868261, 0.14239343, 0.11023512,\n 0.14778395, 0.16164394, 0.10489661, 0.18968627, 0.14316842,\n 0.11805074, 0.22660089, 0.1200424 , 0.10470579, 0.16371943,\n 0.17377892, 0.18715481, 0.22716655, 0.17646216, 0.23168683,\n 0.27621201, 0.24190928, 0.24568164, 0.27721789, 0.18894817,\n 0.17651607, 0.25804242, 0.25466436, 0.24356425, 0.27137646,\n 0.26950634, 0.26841003, 0.20094003, 0.23538765, 0.30279839,\n 0.26703548, 0.33803838, 0.25293982, 0.39510429, 0.32475549,\n 0.37583229, 0.41360974, 0.35663843, 0.36441237, 0.42298791,\n 0.41577956, 0.35225675, 0.35236031, 0.37178853, 0.47746199,\n 0.34010369, 0.42382655, 0.39087629, 0.35522407, 0.41617063,\n 0.40837359, 0.3979013 , 0.43984446, 0.41557294, 0.48908329,\n 0.37935078, 0.45942187, 0.44856036, 0.38298225, 0.51404089,\n 0.4857159 , 0.44875064, 0.48845494, 0.36536103, 0.49337345,\n 0.4878121 , 0.53487682, 0.48096707, 0.48722684, 0.55716145,\n 0.52983743, 0.53781897, 0.48319906, 0.56583619, 0.54767269,\n 0.52520764, 0.61498129, 0.65022099, 0.61341518, 0.57727617,\n 0.64494199, 0.56495082, 0.59615916, 0.55643541, 0.64793307,\n 0.6525358 , 0.68406355, 0.69522488, 0.60621345, 0.62493914,\n 0.68541002, 0.7294538 , 0.6625222 , 0.66217619, 0.73716193,\n 0.63453728, 0.69710797, 0.67195445, 0.74180233, 0.75182641,\n 0.65460467, 0.74208349, 0.68132842, 0.7162168 , 0.80176228,\n 0.74665469, 0.70813477, 0.70945966, 0.69489384, 0.73149681,\n 0.78204733, 0.78100991, 0.79545808, 0.71703029, 0.77955204,\n 0.72948891, 0.79402614, 0.67472208, 0.78351867, 0.84044087,\n 0.7436704 , 0.80226338, 0.72331804, 0.76050156, 0.78629065,\n 0.81037724, 0.84310704, 0.84945536, 0.82753742, 0.82777786,\n 0.83255565, 0.86202121, 0.80740267, 0.7875964 , 0.8151859 ,\n 0.84651113, 0.84920013, 0.89656442, 0.80719292, 0.81870991,\n 0.81100315, 0.8620255 , 0.86308622, 0.86775792, 0.84682673,\n 0.85733885, 0.86692071, 0.91375679, 0.86216998, 0.89378011,\n 0.84808165, 0.88055176, 0.88660747, 0.86753052, 0.80116636,\n 0.83179617, 0.85604507, 0.8775171 , 0.85739475, 0.82735217,\n 0.89296418, 0.90577525, 0.91907412, 0.90477628, 0.88819498,\n 0.84562141, 0.90924245, 0.85359746, 0.83761919, 0.87293065,\n 0.81267947, 0.8590346 , 0.87927586, 0.883394 , 0.84233898,\n 0.88412291, 0.92026424, 0.89491963, 0.89155346, 0.85926378,\n 0.92588449, 0.87111503, 0.89339679, 0.88937104, 0.86701727,\n 0.89090145, 0.89803648, 0.89469522, 0.89581245, 0.86648291,\n 0.9356299 , 0.82345468, 0.89719951, 0.90353125, 0.90820968,\n 0.91956103, 0.91295099, 0.911874 , 0.91836798, 0.92909938,\n 0.9195748 , 0.90447897, 0.91163856, 0.9166432 , 0.90439403,\n 0.93456596, 0.89109629, 0.916596 , 0.89541703, 0.92376363,\n 0.86302412, 0.90646076, 0.91257429, 0.86500078, 0.90097648,\n 0.92672694, 0.90231717, 0.9040935 , 0.88698047, 0.92325824,\n 0.91040045, 0.87112761, 0.89002615, 0.90244317, 0.93935961,\n 0.91013396, 0.93265158, 0.94031835, 0.92176384, 0.91058385,\n 0.92757195, 0.89863437, 0.9056235 , 0.88843673, 0.92200601,\n 0.92987853, 0.92485046, 0.92033291, 0.92019296, 0.91361904,\n 0.91172487, 0.92638123, 0.92463005, 0.9203257 , 0.95217496,\n 0.93138516, 0.9414115 , 0.89878815, 0.91679597, 0.90770209,\n 0.90049917, 0.93927115, 0.91305852, 0.93607372, 0.94809628,\n 0.92966557, 0.93538272, 0.94695014, 0.91146976, 0.91762054,\n 0.9158138 , 0.89066845, 0.94409531, 0.94625849, 0.91788489,\n 0.92335355, 0.93302298, 0.9429673 , 0.90647048, 0.92533755,\n 0.93104702, 0.93519622, 0.92361897, 0.93421835, 0.93407559,\n 0.92425352, 0.94209212, 0.92460293, 0.91686833, 0.93501681,\n 0.92545307, 0.95942056, 0.94846052, 0.9291963 , 0.91908103,\n 0.92965692, 0.93382847, 0.90488452, 0.91060019, 0.91034567,\n 0.93676507, 0.92386204, 0.90883815, 0.9309659 , 0.93630409,\n 0.94661409, 0.8890487 , 0.90644187, 0.94834024, 0.94780028,\n 0.90503323, 0.94931257, 0.90131879, 0.91603428, 0.9345901 ,\n 0.93050313, 0.92168605, 0.92778122, 0.94033247, 0.94720316,\n 0.93285966, 0.94109315, 0.96212828, 0.90340686, 0.91160518,\n 0.94254065, 0.93715638, 0.9184503 , 0.90122396, 0.91595811,\n 0.90234035, 0.90513283, 0.91232675, 0.8908869 , 0.92281044,\n 0.92181432, 0.89422488, 0.93430614, 0.95267183, 0.93316442,\n 0.93459928, 0.92804348, 0.93520498, 0.90923339, 0.94096786,\n 0.88941145, 0.91675609, 0.93678176, 0.93335271, 0.9260872 ,\n 0.89648992, 0.93183416, 0.90977579, 0.92213351, 0.94835836,\n 0.95603991, 0.94048131, 0.92618048, 0.91633701, 0.92115647,\n 0.93254149, 0.92434335, 0.91027969, 0.94091451, 0.91165143,\n 0.93505901, 0.93058741, 0.90304238, 0.95893788, 0.93835539,\n 0.94504249, 0.89894092, 0.92983294, 0.91689557, 0.93456048,\n 0.93991524, 0.92549849, 0.90427381, 0.91659564, 0.92164052,\n 0.91564733, 0.92897671, 0.92515814, 0.9487654 , 0.92560446,\n 0.93244416, 0.93921423, 0.91621238, 0.94594359, 0.91766167,\n 0.93677139, 0.90896547, 0.95078295, 0.92747295, 0.94907326,\n 0.9454596 , 0.93365133, 0.91180551, 0.88942176, 0.95529908,\n 0.9457683 , 0.93159431, 0.93419743, 0.93395591, 0.93568033,\n 0.93305379, 0.91960961, 0.91270286, 0.94664258, 0.92214495,\n 0.94323397, 0.93474811, 0.92038286, 0.91602468, 0.91874373,\n 0.9446891 , 0.89671206, 0.93802589, 0.93421906, 0.93210047,\n 0.932648 , 0.92481577, 0.94111973, 0.95000643, 0.88257021,\n 0.94617885, 0.93924969, 0.96202362, 0.92961437, 0.90242541,\n 0.92824113, 0.95670897, 0.92632753, 0.91165066, 0.92028749,\n 0.94648045, 0.90410364, 0.92212975, 0.8971985 , 0.92386657,\n 0.91137409, 0.9396174 , 0.91514498, 0.95024318, 0.94871414,\n 0.92030919, 0.92560101, 0.93294346, 0.92825991, 0.90771478,\n 0.93846285, 0.93018371, 0.94545019, 0.90936506, 0.93284535,\n 0.93889016, 0.90291345, 0.90938371, 0.91047937, 0.92343152,\n 0.94635969, 0.94280803, 0.9111383 , 0.89855701, 0.9217993 ,\n 0.91408598, 0.94015443, 0.94072664, 0.9363637 , 0.91200626,\n 0.91676378, 0.91757846, 0.93506914, 0.92417806, 0.93227524,\n 0.93286341, 0.93754113, 0.90547085, 0.9208383 , 0.96108609,\n 0.88942879, 0.96143472, 0.89700186, 0.92654228, 0.95694679,\n 0.94653034, 0.9210794 , 0.9394142 , 0.93442118, 0.92785221,\n 0.95501161, 0.87973398, 0.93439466, 0.93263334, 0.95039839,\n 0.91894382, 0.94189674, 0.93663102, 0.94046569, 0.90754175,\n 0.93642449, 0.92648917, 0.91225559, 0.91051596, 0.92513418,\n 0.94030219, 0.92739862, 0.90908056, 0.91720718, 0.9346928 ], dtype=float32)
", "rawMarkdown": "Softmax output. This might be elementary, but still i am having trouble figuring this out. The Mxnet model creates a mx.symbol.LogisticRegressionOutput with a softmax function. I had previously interpreted the output as a probability distribution over the 600 possible outcomes, normalized such that it sums to 1.\r\n\r\nHowever, i might be wrong in my understanding, as the diastole_prob object (the prediction of the model on the validation set) upon inspection of the first row / first observations looks like per below. This row of 600 observations neither is ascending (like a CDF) nor sums to 1. So i am bit lost - can anyone help me and share their perspective? Much appreciated. W \r\n\r\n\r\nIn [16]: diastole_prob[1,:]\r\nOut[16]:\r\narray([ 0.05217434, 0.0474867 , 0.08093406, 0.05017258, 0.06756466,\r\n 0.07113108, 0.05732923, 0.0514702 , 0.07549778, 0.09578432,\r\n 0.06039102, 0.06391794, 0.07802723, 0.06751554, 0.05773542,\r\n 0.0845582 , 0.05266474, 0.06205872, 0.07586135, 0.05797536,\r\n 0.09539511, 0.07857879, 0.0782773 , 0.07885416, 0.07488552,\r\n 0.05688406, 0.04375266, 0.08723985, 0.09554416, 0.13030726,\r\n 0.08048934, 0.07286929, 0.06500199, 0.08452114, 0.05599517,\r\n 0.06534485, 0.12171374, 0.07284499, 0.07555397, 0.05720131,\r\n 0.05926706, 0.08990043, 0.05770419, 0.06937158, 0.10790928,\r\n 0.07708788, 0.13347004, 0.07827558, 0.06507879, 0.07119309,\r\n 0.06509305, 0.07390443, 0.11008534, 0.10490668, 0.08557517,\r\n 0.08783437, 0.07303309, 0.08287172, 0.08533155, 0.10156922,\r\n 0.09635049, 0.08800728, 0.0730741 , 0.08773415, 0.1098076 ,\r\n 0.11432482, 0.07831605, 0.06428145, 0.08088743, 0.09629309,\r\n 0.08714074, 0.04377145, 0.07941946, 0.12478558, 0.10927155,\r\n 0.13440026, 0.14776656, 0.05986561, 0.07887186, 0.10276607,\r\n 0.07350686, 0.13926943, 0.08472204, 0.10816599, 0.08936805,\r\n 0.0781935 , 0.09617887, 0.1250304 , 0.10030739, 0.11530763,\r\n 0.12390044, 0.10760392, 0.1366507 , 0.09015703, 0.14969452,\r\n 0.13367297, 0.17487478, 0.15868261, 0.14239343, 0.11023512,\r\n 0.14778395, 0.16164394, 0.10489661, 0.18968627, 0.14316842,\r\n 0.11805074, 0.22660089, 0.1200424 , 0.10470579, 0.16371943,\r\n 0.17377892, 0.18715481, 0.22716655, 0.17646216, 0.23168683,\r\n 0.27621201, 0.24190928, 0.24568164, 0.27721789, 0.18894817,\r\n 0.17651607, 0.25804242, 0.25466436, 0.24356425, 0.27137646,\r\n 0.26950634, 0.26841003, 0.20094003, 0.23538765, 0.30279839,\r\n 0.26703548, 0.33803838, 0.25293982, 0.39510429, 0.32475549,\r\n 0.37583229, 0.41360974, 0.35663843, 0.36441237, 0.42298791,\r\n 0.41577956, 0.35225675, 0.35236031, 0.37178853, 0.47746199,\r\n 0.34010369, 0.42382655, 0.39087629, 0.35522407, 0.41617063,\r\n 0.40837359, 0.3979013 , 0.43984446, 0.41557294, 0.48908329,\r\n 0.37935078, 0.45942187, 0.44856036, 0.38298225, 0.51404089,\r\n 0.4857159 , 0.44875064, 0.48845494, 0.36536103, 0.49337345,\r\n 0.4878121 , 0.53487682, 0.48096707, 0.48722684, 0.55716145,\r\n 0.52983743, 0.53781897, 0.48319906, 0.56583619, 0.54767269,\r\n 0.52520764, 0.61498129, 0.65022099, 0.61341518, 0.57727617,\r\n 0.64494199, 0.56495082, 0.59615916, 0.55643541, 0.64793307,\r\n 0.6525358 , 0.68406355, 0.69522488, 0.60621345, 0.62493914,\r\n 0.68541002, 0.7294538 , 0.6625222 , 0.66217619, 0.73716193,\r\n 0.63453728, 0.69710797, 0.67195445, 0.74180233, 0.75182641,\r\n 0.65460467, 0.74208349, 0.68132842, 0.7162168 , 0.80176228,\r\n 0.74665469, 0.70813477, 0.70945966, 0.69489384, 0.73149681,\r\n 0.78204733, 0.78100991, 0.79545808, 0.71703029, 0.77955204,\r\n 0.72948891, 0.79402614, 0.67472208, 0.78351867, 0.84044087,\r\n 0.7436704 , 0.80226338, 0.72331804, 0.76050156, 0.78629065,\r\n 0.81037724, 0.84310704, 0.84945536, 0.82753742, 0.82777786,\r\n 0.83255565, 0.86202121, 0.80740267, 0.7875964 , 0.8151859 ,\r\n 0.84651113, 0.84920013, 0.89656442, 0.80719292, 0.81870991,\r\n 0.81100315, 0.8620255 , 0.86308622, 0.86775792, 0.84682673,\r\n 0.85733885, 0.86692071, 0.91375679, 0.86216998, 0.89378011,\r\n 0.84808165, 0.88055176, 0.88660747, 0.86753052, 0.80116636,\r\n 0.83179617, 0.85604507, 0.8775171 , 0.85739475, 0.82735217,\r\n 0.89296418, 0.90577525, 0.91907412, 0.90477628, 0.88819498,\r\n 0.84562141, 0.90924245, 0.85359746, 0.83761919, 0.87293065,\r\n 0.81267947, 0.8590346 , 0.87927586, 0.883394 , 0.84233898,\r\n 0.88412291, 0.92026424, 0.89491963, 0.89155346, 0.85926378,\r\n 0.92588449, 0.87111503, 0.89339679, 0.88937104, 0.86701727,\r\n 0.89090145, 0.89803648, 0.89469522, 0.89581245, 0.86648291,\r\n 0.9356299 , 0.82345468, 0.89719951, 0.90353125, 0.90820968,\r\n 0.91956103, 0.91295099, 0.911874 , 0.91836798, 0.92909938,\r\n 0.9195748 , 0.90447897, 0.91163856, 0.9166432 , 0.90439403,\r\n 0.93456596, 0.89109629, 0.916596 , 0.89541703, 0.92376363,\r\n 0.86302412, 0.90646076, 0.91257429, 0.86500078, 0.90097648,\r\n 0.92672694, 0.90231717, 0.9040935 , 0.88698047, 0.92325824,\r\n 0.91040045, 0.87112761, 0.89002615, 0.90244317, 0.93935961,\r\n 0.91013396, 0.93265158, 0.94031835, 0.92176384, 0.91058385,\r\n 0.92757195, 0.89863437, 0.9056235 , 0.88843673, 0.92200601,\r\n 0.92987853, 0.92485046, 0.92033291, 0.92019296, 0.91361904,\r\n 0.91172487, 0.92638123, 0.92463005, 0.9203257 , 0.95217496,\r\n 0.93138516, 0.9414115 , 0.89878815, 0.91679597, 0.90770209,\r\n 0.90049917, 0.93927115, 0.91305852, 0.93607372, 0.94809628,\r\n 0.92966557, 0.93538272, 0.94695014, 0.91146976, 0.91762054,\r\n 0.9158138 , 0.89066845, 0.94409531, 0.94625849, 0.91788489,\r\n 0.92335355, 0.93302298, 0.9429673 , 0.90647048, 0.92533755,\r\n 0.93104702, 0.93519622, 0.92361897, 0.93421835, 0.93407559,\r\n 0.92425352, 0.94209212, 0.92460293, 0.91686833, 0.93501681,\r\n 0.92545307, 0.95942056, 0.94846052, 0.9291963 , 0.91908103,\r\n 0.92965692, 0.93382847, 0.90488452, 0.91060019, 0.91034567,\r\n 0.93676507, 0.92386204, 0.90883815, 0.9309659 , 0.93630409,\r\n 0.94661409, 0.8890487 , 0.90644187, 0.94834024, 0.94780028,\r\n 0.90503323, 0.94931257, 0.90131879, 0.91603428, 0.9345901 ,\r\n 0.93050313, 0.92168605, 0.92778122, 0.94033247, 0.94720316,\r\n 0.93285966, 0.94109315, 0.96212828, 0.90340686, 0.91160518,\r\n 0.94254065, 0.93715638, 0.9184503 , 0.90122396, 0.91595811,\r\n 0.90234035, 0.90513283, 0.91232675, 0.8908869 , 0.92281044,\r\n 0.92181432, 0.89422488, 0.93430614, 0.95267183, 0.93316442,\r\n 0.93459928, 0.92804348, 0.93520498, 0.90923339, 0.94096786,\r\n 0.88941145, 0.91675609, 0.93678176, 0.93335271, 0.9260872 ,\r\n 0.89648992, 0.93183416, 0.90977579, 0.92213351, 0.94835836,\r\n 0.95603991, 0.94048131, 0.92618048, 0.91633701, 0.92115647,\r\n 0.93254149, 0.92434335, 0.91027969, 0.94091451, 0.91165143,\r\n 0.93505901, 0.93058741, 0.90304238, 0.95893788, 0.93835539,\r\n 0.94504249, 0.89894092, 0.92983294, 0.91689557, 0.93456048,\r\n 0.93991524, 0.92549849, 0.90427381, 0.91659564, 0.92164052,\r\n 0.91564733, 0.92897671, 0.92515814, 0.9487654 , 0.92560446,\r\n 0.93244416, 0.93921423, 0.91621238, 0.94594359, 0.91766167,\r\n 0.93677139, 0.90896547, 0.95078295, 0.92747295, 0.94907326,\r\n 0.9454596 , 0.93365133, 0.91180551, 0.88942176, 0.95529908,\r\n 0.9457683 , 0.93159431, 0.93419743, 0.93395591, 0.93568033,\r\n 0.93305379, 0.91960961, 0.91270286, 0.94664258, 0.92214495,\r\n 0.94323397, 0.93474811, 0.92038286, 0.91602468, 0.91874373,\r\n 0.9446891 , 0.89671206, 0.93802589, 0.93421906, 0.93210047,\r\n 0.932648 , 0.92481577, 0.94111973, 0.95000643, 0.88257021,\r\n 0.94617885, 0.93924969, 0.96202362, 0.92961437, 0.90242541,\r\n 0.92824113, 0.95670897, 0.92632753, 0.91165066, 0.92028749,\r\n 0.94648045, 0.90410364, 0.92212975, 0.8971985 , 0.92386657,\r\n 0.91137409, 0.9396174 , 0.91514498, 0.95024318, 0.94871414,\r\n 0.92030919, 0.92560101, 0.93294346, 0.92825991, 0.90771478,\r\n 0.93846285, 0.93018371, 0.94545019, 0.90936506, 0.93284535,\r\n 0.93889016, 0.90291345, 0.90938371, 0.91047937, 0.92343152,\r\n 0.94635969, 0.94280803, 0.9111383 , 0.89855701, 0.9217993 ,\r\n 0.91408598, 0.94015443, 0.94072664, 0.9363637 , 0.91200626,\r\n 0.91676378, 0.91757846, 0.93506914, 0.92417806, 0.93227524,\r\n 0.93286341, 0.93754113, 0.90547085, 0.9208383 , 0.96108609,\r\n 0.88942879, 0.96143472, 0.89700186, 0.92654228, 0.95694679,\r\n 0.94653034, 0.9210794 , 0.9394142 , 0.93442118, 0.92785221,\r\n 0.95501161, 0.87973398, 0.93439466, 0.93263334, 0.95039839,\r\n 0.91894382, 0.94189674, 0.93663102, 0.94046569, 0.90754175,\r\n 0.93642449, 0.92648917, 0.91225559, 0.91051596, 0.92513418,\r\n 0.94030219, 0.92739862, 0.90908056, 0.91720718, 0.9346928 ], dtype=float32)" }, { "id": 105943, "postDate": "2016-01-27T15:21:07.197Z", "content": "[quote=Kosiński KM;105942]
\n\nGuys,
\n\nHow long (in terms of computing time) does it typically take you to go through the pre-processing step of this tutorial? I left the script running over night, but it wasn't finished when I woke up. I expected it to run for quite some time, but not that long to be honest...
\n\nCheers,\nK
\n\n[/quote]\nthe preprocess script should not run longer than 1 hour, no matter SSD or spin harddrive. it takes about 2x CPUs on my machine. if your top shows a python process with about 200% CPU, it is normal, otherwise, there are something weird.
Guys,
\n\nHow long (in terms of computing time) does it typically take you to go through the pre-processing step of this tutorial? I left the script running over night, but it wasn't finished when I woke up. I expected it to run for quite some time, but not that long to be honest...
\n\nCheers,\nK
", "rawMarkdown": "Guys,\r\n\r\nHow long (in terms of computing time) does it typically take you to go through the pre-processing step of this tutorial? I left the script running over night, but it wasn't finished when I woke up. I expected it to run for quite some time, but not that long to be honest...\r\n\r\nCheers,\r\nK" }, { "id": 105873, "postDate": "2016-01-27T05:27:28.940Z", "content": "@EIGSI, I am seeing the same issue when using the training data also as the validation data.
\n\nLooking at _train_multi_device in model.py it looks like the training data metric (eval_metric) is an accumulation of the error of each batch during the epoch. During the epoch, for each batch there is also error correction happening so the value reported is not equivalent to what you would get if you passed the training data through the network at the end of the epoch AFTER all error correction.
\n\nFrom what I can determine in the code, using the training data as the validation data gives you an accurate assessment of the training error at the end of the epoch.
", "rawMarkdown": "@EIGSI, I am seeing the same issue when using the training data also as the validation data. \r\n\r\nLooking at _train_multi_device in model.py it looks like the training data metric (eval_metric) is an accumulation of the error of each batch during the epoch. During the epoch, for each batch there is also error correction happening so the value reported is not equivalent to what you would get if you passed the training data through the network at the end of the epoch AFTER all error correction.\r\n\r\nFrom what I can determine in the code, using the training data as the validation data gives you an accurate assessment of the training error at the end of the epoch." }, { "id": 105815, "postDate": "2016-01-26T22:02:28.440Z", "content": "@EIGSI. that seems strange. however - potentially the files start with a different random initialization of the weights, and subsequently might show slightly different CPRS values. if the regularizatoin, learning rate and wd are appropriate, then both models should converge in the same direction and on the same values. I can imagine that if the models are overfitting / not converging, then that might go off in very different reactions. I might be wrong though.
", "rawMarkdown": "@EIGSI. that seems strange. however - potentially the files start with a different random initialization of the weights, and subsequently might show slightly different CPRS values. if the regularizatoin, learning rate and wd are appropriate, then both models should converge in the same direction and on the same values. I can imagine that if the models are overfitting / not converging, then that might go off in very different reactions. I might be wrong though. " }, { "id": 105797, "postDate": "2016-01-26T19:42:10.157Z", "content": "@Bing / MXNet,
\n\nThe MXNet git hub said they'd be posting the pre-processing file written in R. Is there any chance that is coming out soon?
\n\nThanks!
", "rawMarkdown": "@Bing / MXNet,\r\n\r\nThe MXNet git hub said they'd be posting the pre-processing file written in R. Is there any chance that is coming out soon?\r\n\r\nThanks!" }, { "id": 105794, "postDate": "2016-01-26T19:26:58.433Z", "content": "In this tutorial, if I specify the same file for local validation as the local training file, I am getting totally different CRPS at each epoch, since the data files are same, should not the CRPS be same? Is it an mxnet bug or am I missing something?
", "rawMarkdown": "In this tutorial, if I specify the same file for local validation as the local training file, I am getting totally different CRPS at each epoch, since the data files are same, should not the CRPS be same? Is it an mxnet bug or am I missing something?" }, { "id": 105678, "postDate": "2016-01-25T19:47:54.580Z", "content": "@WD, thanks for the reference. Using the eval_data is exactly what I was missing.
", "rawMarkdown": "@WD, thanks for the reference. Using the eval_data is exactly what I was missing." }, { "id": 105677, "postDate": "2016-01-25T19:46:31.750Z", "content": "Bing,\nI am looking forward for tutorial, tried subclass cvsiter but that does not look like a standalone class also\nTried mxrecordio as suggested in issue #1332 and sframe however samples are so limited that i failed. only option left to Rewrite preprocessing for imrec (not yet attempted). Any simpler solution using cvs\nCould be great.\nThanks
\n\n[quote=Bing Xu;105656]
\n\nThe easiest should use SFrame.\nI will write a tutorial on how to use SFrame with MXNet. Hopefully in this week.\nhttps://github.com/dmlc/mxnet/tree/master/plugin/sframe
\n\nhttps://github.com/dato-code/SFrame
\n\n[quote=WD;105654]
\n\n@phunter. thanks for this. i will look into both the CSV route and the Io.record route. in the CSV approach - i guess the easiest way would be to work with the numpy "img" object - no? i will play around with this and make code public if i find something useful to others
\n\n f = dicom.read_file(path)\n img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\n dst_path = path.rsplit(".", 1)[0] + ".64x64.jpg" #create jpg out of all files\n[/quote]
\n\n[/quote]
", "rawMarkdown": "Bing,\r\nI am looking forward for tutorial, tried subclass cvsiter but that does not look like a standalone class also\r\nTried mxrecordio as suggested in issue #1332 and sframe however samples are so limited that i failed. only option left to Rewrite preprocessing for imrec (not yet attempted). Any simpler solution using cvs\r\nCould be great.\r\nThanks\r\n\r\n[quote=Bing Xu;105656]\r\n\r\nThe easiest should use SFrame.\r\nI will write a tutorial on how to use SFrame with MXNet. Hopefully in this week.\r\nhttps://github.com/dmlc/mxnet/tree/master/plugin/sframe\r\n\r\nhttps://github.com/dato-code/SFrame\r\n\r\n\r\n[quote=WD;105654]\r\n\r\n@phunter. thanks for this. i will look into both the CSV route and the Io.record route. in the CSV approach - i guess the easiest way would be to work with the numpy \"img\" object - no? i will play around with this and make code public if i find something useful to others\r\n\r\n f = dicom.read_file(path)\r\n img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\r\n dst_path = path.rsplit(\".\", 1)[0] + \".64x64.jpg\" #create jpg out of all files\r\n\r\n[/quote]\r\n\r\n[/quote]\r\n" }, { "id": 105670, "postDate": "2016-01-25T19:07:43.987Z", "content": "@athyssen. i copied in the code that i use on page 7 of this thread. let me know if that helps
", "rawMarkdown": "@athyssen. i copied in the code that i use on page 7 of this thread. let me know if that helps" }, { "id": 105667, "postDate": "2016-01-25T18:47:23.650Z", "content": "If I have a training dataset and testing dataset with proper labels for each, what is the suggested method for logging both the training score (currently seeing Train-CRPS-xxxx) and also an independent score for the testing dataset. I need to watch for overtraining and diverging CRPS scores for the two datasets.
\n\nCurrently, I have implemented an epoch_end_callback where I load the test input/labels and use:\ntest_pred = model.predict(test_data), followed by calculating the CRPS for the test predictions. Although I see the training score changing the CRPS of the test predictions remains the same. If I print the test_pred output, the values are the same for each epoch.
\n\nThanks!
", "rawMarkdown": "If I have a training dataset and testing dataset with proper labels for each, what is the suggested method for logging both the training score (currently seeing Train-CRPS-xxxx) and also an independent score for the testing dataset. I need to watch for overtraining and diverging CRPS scores for the two datasets.\r\n\r\nCurrently, I have implemented an epoch_end_callback where I load the test input/labels and use:\r\ntest_pred = model.predict(test_data), followed by calculating the CRPS for the test predictions. Although I see the training score changing the CRPS of the test predictions remains the same. If I print the test_pred output, the values are the same for each epoch. \r\n\r\nThanks!" }, { "id": 105653, "postDate": "2016-01-25T16:45:30.057Z", "content": "Do we have some (another) tutorial of using imageRecordIter for this competition? It may also help the IO speed of feeding data to GPU.
", "rawMarkdown": "Do we have some (another) tutorial of using imageRecordIter for this competition? It may also help the IO speed of feeding data to GPU." }, { "id": 105650, "postDate": "2016-01-25T16:33:32.493Z", "content": "the mxnet tutorial code has not (yet) support image rotation. one can modify preprocessing.py and generate CSV with rotation by scikit-image
", "rawMarkdown": "the mxnet tutorial code has not (yet) support image rotation. one can modify preprocessing.py and generate CSV with rotation by scikit-image" }, { "id": 105648, "postDate": "2016-01-25T16:30:11.643Z", "content": "@JuGL - how did you do the rotation in Mxnet?
", "rawMarkdown": "@JuGL - how did you do the rotation in Mxnet? " }, { "id": 105636, "postDate": "2016-01-25T14:02:12.270Z", "content": "I did not use the image difference approach, just some preprocessing like resampling and rotation. @DavidGbodiOdaibo
", "rawMarkdown": "I did not use the image difference approach, just some preprocessing like resampling and rotation. @DavidGbodiOdaibo" }, { "id": 105629, "postDate": "2016-01-25T12:45:28.827Z", "content": "@JuGL ..cool! I am not using the Mxnet...but are you using the image difference approach in the tutorial or some other preprocessing? I would like to know if the secret to the low scores is the image diff. I am using canny edge detection for preprocessing and noise reduction. I tried the image diff and visualized the result of the diff, it looked like a lot of noise and I could barely make out any structure from the original image so I decided to scrap it.
", "rawMarkdown": "@JuGL ..cool! I am not using the Mxnet...but are you using the image difference approach in the tutorial or some other preprocessing? I would like to know if the secret to the low scores is the image diff. I am using canny edge detection for preprocessing and noise reduction. I tried the image diff and visualized the result of the diff, it looked like a lot of noise and I could barely make out any structure from the original image so I decided to scrap it." }, { "id": 105621, "postDate": "2016-01-25T10:16:24.163Z", "content": "Rotation. Has anyone figured out how do rotations when using the mxnet.io.CSVIter? The documentaiton shows that one can apply rotation when using mxnet.io.ImageRecordIter, but i am not sure how one can apply this to CSVIter. Any thoughts or help much appreciated.
", "rawMarkdown": "Rotation. Has anyone figured out how do rotations when using the mxnet.io.CSVIter? The documentaiton shows that one can apply rotation when using mxnet.io.ImageRecordIter, but i am not sure how one can apply this to CSVIter. Any thoughts or help much appreciated." }, { "id": 105546, "postDate": "2016-01-24T15:32:28.943Z", "content": "Early stopping. has anyone already coded an early stopping rule for Mxnet training in Python. I am thinking of adapting http://mxnet.readthedocs.org/en/latest/R-package/CallbackFunctionTutorial.html - but I was wondering if someone has already done this - and would be willing to share this? Many thanks in advance, let me know. If not - then i will give this a try and share on this forum. W
", "rawMarkdown": "Early stopping. has anyone already coded an early stopping rule for Mxnet training in Python. I am thinking of adapting http://mxnet.readthedocs.org/en/latest/R-package/CallbackFunctionTutorial.html - but I was wondering if someone has already done this - and would be willing to share this? Many thanks in advance, let me know. If not - then i will give this a try and share on this forum. W " }, { "id": 105505, "postDate": "2016-01-23T20:50:54.660Z", "content": "The following code is a very rudimentary way (i am sure there are cleaner and more elegant ways) to highlight the number of files per directory and to isolate those directories that have fewer than 30 files. Ubuntu - and it saves the output the logfilestructure file
\n\nfind . -type d -print0 | while read -d '' -r dir; do files=("$dir"/*); printf "%5d files in directory %s\\n" "${#files[@]}" "$dir" ; done 2>&1 | tee logfilestructure
", "rawMarkdown": "The following code is a very rudimentary way (i am sure there are cleaner and more elegant ways) to highlight the number of files per directory and to isolate those directories that have fewer than 30 files. Ubuntu - and it saves the output the logfilestructure file\r\n\r\nfind . -type d -print0 | while read -d '' -r dir; do files=(\"$dir\"/*); printf \"%5d files in directory %s\\n\" \"${#files[@]}\" \"$dir\" ; done 2>&1 | tee logfilestructure " }, { "id": 105504, "postDate": "2016-01-23T20:44:43.147Z", "content": "@WD - the additional data improved my score.
", "rawMarkdown": "@WD - the additional data improved my score." }, { "id": 105503, "postDate": "2016-01-23T20:37:58.083Z", "content": "@DavidGbodiOdaibo - this is an intriguing suggestion. Have you observed that adding this incremental data improves performance?
", "rawMarkdown": "@DavidGbodiOdaibo - this is an intriguing suggestion. Have you observed that adding this incremental data improves performance? " }, { "id": 105493, "postDate": "2016-01-23T19:02:53.813Z", "content": "@WD: I scale the size of a cropped section by dicom pixel spacing and slice width to make each pixel represent a constant volume element. I've spent all my time preprocessing and haven't trained with all this yet. So a big heart is directly represented in the image.
", "rawMarkdown": "@WD: I scale the size of a cropped section by dicom pixel spacing and slice width to make each pixel represent a constant volume element. I've spent all my time preprocessing and haven't trained with all this yet. So a big heart is directly represented in the image." }, { "id": 105488, "postDate": "2016-01-23T18:08:06.183Z", "content": "@WD if stack < 30, duplicate the last image in stack until 30. Caution if you are using the image diff approach this will produce a black image for the duplicated images.
", "rawMarkdown": "@WD if stack < 30, duplicate the last image in stack until 30. Caution if you are using the image diff approach this will produce a black image for the duplicated images." }, { "id": 105487, "postDate": "2016-01-23T18:02:25.960Z", "content": "Net configurations. On a diferent topics - and to help others - i did not so far get improved results in comparison to the original train.py file in making changes to the type of activiation function, the size of the filters, the number of filters and other incremental changes. I am happy to share the validation curves and more detailed results- and interested to hear if somebody else has obtained different results. I tried as well to apply the out-of-the-box google-net with minor adjustments - and so far have not seen good performance (although this might be a tuning question).
\n\nI am currently thinking on how to incorporate the information from those stacks with e.g. 29 images or 25 images. I don't think there is a straightforward way of doing this (besides training seperate networks for 30-stacks, 29-stacks, etc.) - but would love to hear if there is a smarter / more elegant way of doing this.
", "rawMarkdown": "Net configurations. On a diferent topics - and to help others - i did not so far get improved results in comparison to the original train.py file in making changes to the type of activiation function, the size of the filters, the number of filters and other incremental changes. I am happy to share the validation curves and more detailed results- and interested to hear if somebody else has obtained different results. I tried as well to apply the out-of-the-box google-net with minor adjustments - and so far have not seen good performance (although this might be a tuning question). \r\n\r\nI am currently thinking on how to incorporate the information from those stacks with e.g. 29 images or 25 images. I don't think there is a straightforward way of doing this (besides training seperate networks for 30-stacks, 29-stacks, etc.) - but would love to hear if there is a smarter / more elegant way of doing this." }, { "id": 105485, "postDate": "2016-01-23T17:47:03.740Z", "content": "@Scott. thanks for sharing this! I am still trying and pondering, in order to open up the "black box" of a CNN on the difference vectors, on what type of features would be relevant for low-volume predictions and what kind of features for high volume predictions. i guess that the curvature of edge features might be linked to the size of the chamber (faint curve -> big heart)? any thoughts / ideas / musings much appreciated!
", "rawMarkdown": "@Scott. thanks for sharing this! I am still trying and pondering, in order to open up the \"black box\" of a CNN on the difference vectors, on what type of features would be relevant for low-volume predictions and what kind of features for high volume predictions. i guess that the curvature of edge features might be linked to the size of the chamber (faint curve -> big heart)? any thoughts / ideas / musings much appreciated! " }, { "id": 105478, "postDate": "2016-01-23T15:28:05.760Z", "content": "[quote=WD;105472]\nHi all. I am trying to visualize / understand how the features might look that the algorithm is learning from the "difference vector" that we feed into the mxnet algo in the mxnet tutoral (e.g. diffs = [frames[i+1] - frames[i] for i in range(29)])
\n\nFor a "normal" CNN where we insert the actual pixels (rather than the differences between frames) - i understand that the features are increasingly more complex features of an image as we proceed through the layers (e.g. edge -> curve -> part of a wheel, etc.). When we think of a difference vector - what would be the types of features that the CNN is learning? is there a way that we can visualize the weights / these features in an easy and intuitive manner?
\n\nAny thoughts much appreciated, W \n[/quote]
\n\n@WD: Thought I was being original and hadn't dug much into the tutorial, but I thought it made sense to look at the frame differences as a measure of what has changed in the image. It looked like the heart should be changing most in the time scale we are considering. The attached image is output of some preprocessing script that I'm using to locate the heart and then in this case plot what's changing. If the heart wasn't beating, the image array would be uniformly valued.
\n\nScott
", "rawMarkdown": "[quote=WD;105472]\r\nHi all. I am trying to visualize / understand how the features might look that the algorithm is learning from the \"difference vector\" that we feed into the mxnet algo in the mxnet tutoral (e.g. diffs = [frames[i+1] - frames[i] for i in range(29)]) \r\n\r\nFor a \"normal\" CNN where we insert the actual pixels (rather than the differences between frames) - i understand that the features are increasingly more complex features of an image as we proceed through the layers (e.g. edge -> curve -> part of a wheel, etc.). When we think of a difference vector - what would be the types of features that the CNN is learning? is there a way that we can visualize the weights / these features in an easy and intuitive manner? \r\n\r\nAny thoughts much appreciated, W \r\n[/quote]\r\n\r\n@WD: Thought I was being original and hadn't dug much into the tutorial, but I thought it made sense to look at the frame differences as a measure of what has changed in the image. It looked like the heart should be changing most in the time scale we are considering. The attached image is output of some preprocessing script that I'm using to locate the heart and then in this case plot what's changing. If the heart wasn't beating, the image array would be uniformly valued.\r\n\r\nScott\r\n\r\n\r\n" }, { "id": 105476, "postDate": "2016-01-23T15:09:24.720Z", "content": "@Jon: I used the command 'pip install pydicom' at the anaconda prompt in windows and then the same in the linux terminal with anaconda and Python 3 installed in both instances. If you're not doing it already, I recommend using Anaconda to manage Python in Windows and like it in linux also. I don't have a great deal of experience and it simplifies things.
", "rawMarkdown": "@Jon: I used the command 'pip install pydicom' at the anaconda prompt in windows and then the same in the linux terminal with anaconda and Python 3 installed in both instances. If you're not doing it already, I recommend using Anaconda to manage Python in Windows and like it in linux also. I don't have a great deal of experience and it simplifies things." }, { "id": 105472, "postDate": "2016-01-23T13:02:12.930Z", "content": "Hi all. I am trying to visualize / understand how the features might look that the algorithm is learning from the "difference vector" that we feed into the mxnet algo in the mxnet tutoral (e.g. diffs = [frames[i+1] - frames[i] for i in range(29)])
\n\nFor a "normal" CNN where we insert the actual pixels (rather than the differences between frames) - i understand that the features are increasingly more complex features of an image as we proceed through the layers (e.g. edge -> curve -> part of a wheel, etc.). When we think of a difference vector - what would be the types of features that the CNN is learning? is there a way that we can visualize the weights / these features in an easy and intuitive manner?
\n\nAny thoughts much appreciated, W
", "rawMarkdown": "Hi all. I am trying to visualize / understand how the features might look that the algorithm is learning from the \"difference vector\" that we feed into the mxnet algo in the mxnet tutoral (e.g. diffs = [frames[i+1] - frames[i] for i in range(29)]) \r\n\r\nFor a \"normal\" CNN where we insert the actual pixels (rather than the differences between frames) - i understand that the features are increasingly more complex features of an image as we proceed through the layers (e.g. edge -> curve -> part of a wheel, etc.). When we think of a difference vector - what would be the types of features that the CNN is learning? is there a way that we can visualize the weights / these features in an easy and intuitive manner? \r\n\r\nAny thoughts much appreciated, W \r\n" }, { "id": 105454, "postDate": "2016-01-23T04:25:42.280Z", "content": "Thanks so much. I am a newbie in machine learning and have spent a lot of time trying this tutorial. Here is my modified python file that could run pre-processing without problem in my Spyder, Python 3.4, as well as the result, which I believe right. Hope this helps others.\nhttps://drive.google.com/folderview?id=0B-VsxhvptLPNUTdFZEdXLWwxNGM&usp=sharing
", "rawMarkdown": "Thanks so much. I am a newbie in machine learning and have spent a lot of time trying this tutorial. Here is my modified python file that could run pre-processing without problem in my Spyder, Python 3.4, as well as the result, which I believe right. Hope this helps others.\r\nhttps://drive.google.com/folderview?id=0B-VsxhvptLPNUTdFZEdXLWwxNGM&usp=sharing " }, { "id": 105401, "postDate": "2016-01-22T21:09:48.803Z", "content": "[quote=Scott Smith;104906]
\n\n@Jon: I think you should have the same number of rows. When I was running into this, I just opened the csv's in notepad++ and visually inspected. You should not have an empty row at the end of any of the csv's, double entries (e.g. in train-label) or empty rows elsewhere. I think the number of rows for train-label and the training vectors should match.
\n\nThe approach I ended up taking though was to just get rid of Windows and move my laptop to Ubuntu 15.04 + anaconda3 + pydicom + CUDA 7.5 + CUDNN 3 + openblas + opencv + mxnet. Having no experience with linux, being a mechanical engineer and really only focusing on R for this late-night hobby, it took a solid week to get through.
\n\n[/quote]
\n\n@Scott did you get pydicom working with the latest version of Python? I get an error with import pydicom and the only thing I can think of is that pydicom only works with earlier versions of Python.
", "rawMarkdown": "[quote=Scott Smith;104906]\r\n\r\n@Jon: I think you should have the same number of rows. When I was running into this, I just opened the csv's in notepad++ and visually inspected. You should not have an empty row at the end of any of the csv's, double entries (e.g. in train-label) or empty rows elsewhere. I think the number of rows for train-label and the training vectors should match.\r\n\r\nThe approach I ended up taking though was to just get rid of Windows and move my laptop to Ubuntu 15.04 + anaconda3 + pydicom + CUDA 7.5 + CUDNN 3 + openblas + opencv + mxnet. Having no experience with linux, being a mechanical engineer and really only focusing on R for this late-night hobby, it took a solid week to get through.\r\n\r\n[/quote]\r\n\r\n@Scott did you get pydicom working with the latest version of Python? I get an error with import pydicom and the only thing I can think of is that pydicom only works with earlier versions of Python." }, { "id": 105165, "postDate": "2016-01-20T15:52:24.853Z", "content": "Thanks WD, I was working on looking into something similar and this should save me time. Thank you so much for sharing! Hopefully I can repay the favor someday.
", "rawMarkdown": " Thanks WD, I was working on looking into something similar and this should save me time. Thank you so much for sharing! Hopefully I can repay the favor someday." }, { "id": 105161, "postDate": "2016-01-20T15:24:24.477Z", "content": "@WD thanks for the advice. Do you have any insight into how much it would cost to run through this exercise (once) using AWS instance? I hesitated to use AWS because I didn't know how much I was in for.
", "rawMarkdown": "@WD thanks for the advice. Do you have any insight into how much it would cost to run through this exercise (once) using AWS instance? I hesitated to use AWS because I didn't know how much I was in for. " }, { "id": 105155, "postDate": "2016-01-20T14:49:55.343Z", "content": "@Jon. you might want to enable GPU - or - probably easier - launch an AWS instance.
", "rawMarkdown": "@Jon. you might want to enable GPU - or - probably easier - launch an AWS instance. " }, { "id": 105154, "postDate": "2016-01-20T14:46:56.397Z", "content": "I am on # In[4]: in Train.py and it is now going on 14 hours and my PC is still chugging away on fitting the stytole model. Currently on Epoc 53. Is there anything I can do outside of buying a new computer to speed this up?
", "rawMarkdown": "I am on # In[4]: in Train.py and it is now going on 14 hours and my PC is still chugging away on fitting the stytole model. Currently on Epoc 53. Is there anything I can do outside of buying a new computer to speed this up?" }, { "id": 105140, "postDate": "2016-01-20T12:54:07.363Z", "content": "Apologies - for some reason it seems to bunch all code in a block. tried different options - but could not figure out how to create the one-line-per-line-of-code formatting. let me know and i will adjust
", "rawMarkdown": "Apologies - for some reason it seems to bunch all code in a block. tried different options - but could not figure out how to create the one-line-per-line-of-code formatting. let me know and i will adjust" }, { "id": 105139, "postDate": "2016-01-20T12:51:37.213Z", "content": "hi all, please find some code below that should help to plot training and validation curves. please note that the graphs are still pretty poorly formatted (I am not that familiar with matplotlib) - but hopefully this will help others. let me know any questions
\n\n1- one has to adjust the train.py and preprocessing scripts to create the validation set. Please note that the naming might be somewhat confusing: in the mxnet script the validation set is the held-out dataset (often in other context referred as the test set) and the cross-validation set is called the test set.
\n\n1A. You want to ensure the the last lines in the preprocessing script are active - and that the training set is split in a local-train and a test set.
\n\n1B. In train.py - you want to add text that creates a data_test file - e.g.
\n\ndata_test = mx.io.CSVIter(data_csv="./test-64x64-data.csv", data_shape=(30, 64, 64),\n label_csv="./test-stytole.csv", label_shape=(600,),\n batch_size=batch_size)
\n\n1C. in train.py - you then want to adjust the model.fit by adding a eval_data paramer - e.g.:
\n\nstytole_model.fit(X=data_train, eval_data=(data_test), eval_metric = mx.metric.np(CRPS))
\n\n1D. (repeat for the diastole model)
\n\nThe following steps are written for AWS users. Users who run the model locally might have to adapt this slightly:
\n\n1E. You want to run the model in a way that logs the output in a seperate file. I am using AWS - so at the command line i run something like: python train.py 2>&1 | tee log
\n\n1F. Subsequently - i will transfer the file to my local environment (e.g. using WinSCP) and use my local python editor (e.g. Spyder) to create a plot with the code per below. This code, and the plots, can clearly be optimized - please share better code / improvements! let me know any questions
\n\nimport matplotlib\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport re
\n\nlog = open('log').read()
\n\nlog_tr = [float(x) for x in TR_RE.findall(log)]\nlog_va = [float(x) for x in VA_RE.findall(log)]\nidx=np.arange(len(log_tr))\nend = len(log_tr)\nmid=end/2
\n\nfig = plt.figure()
\nfig.subplots_adjust(hspace=0.6, wspace=0.8)\nplt.title("")\nax1=fig.add_subplot(221)\nax1.plot(idx[0:mid], log_tr[0:mid], '-', color="r",label="log")\nax1.set_title('train- systole')\nax1.set_ylabel("error")\nax1.set_xlabel("epoch")\nax2=fig.add_subplot(222)\nax2.plot(idx[(mid+1):end], log_tr[(mid+1):end], '-', color="r",label="log")\nax2.set_title('train - diastole')\nax2.set_ylabel("error")\nax2.set_xlabel("epoch")\nax3=fig.add_subplot(223)\nax3.plot(idx[0:mid], log_va[0:mid], '-', color="r",label="log")\nax3.set_title('val - systole')\nax3.set_ylabel("error")\nax3.set_xlabel("epoch")\nax4=fig.add_subplot(224)\nax4.plot(idx[(mid+1):end], log_va[(mid+1):end], '-', color="r",label="log")\nax4.set_title('val - diastole')\nax4.set_ylabel("error")\nax4.set_xlabel("epoch")\nplt.show()
I just extended a tool already available in mxnet (parse_log.py) to plot the progress/score after the training. See the attached file.
", "rawMarkdown": "I just extended a tool already available in mxnet ([parse_log.py][1]) to plot the progress/score after the training. See the attached file.\r\n\r\n [1]: https://github.com/dmlc/mxnet/blob/master/tools/parse_log.py" }, { "id": 105074, "postDate": "2016-01-19T19:08:14.010Z", "content": "[quote=WD;105066]
\n\nI am finishing up code to extract and plot training and validation curves, and I will share this code today or tomorrow. I had a quick question. If one uses a terminal mode to access one's AWS instance (e.g. using Putty) and either runs scripts from the bash command line or from an ipython shell, then is it possible to plot graphs visually? or does one either a) have to install an ipython notebook, or b) save-to-image, transfer the files back to one's local computer, and plot locally?
\n\n[/quote]\nfor option b, you can have
\n\nimport matplotlib as mpl\nmpl.use('Agg') #without X server\n#plot something here\npl.savefig('my_figure.png',bbox_inches="tight")\nWhen I train diastole network on this tutorial, the results are about 0.01 worse than that of systole. Did anyone else notice this?
", "rawMarkdown": "When I train diastole network on this tutorial, the results are about 0.01 worse than that of systole. Did anyone else notice this? " }, { "id": 105053, "postDate": "2016-01-19T14:05:16.527Z", "content": "[quote=kagglekaggle;105052]
\n\nHello all,
\n\nI am a new B here and simulating this e2e tutorial. So far everything goes fine except for \nMiss: 590_Diastole\nMiss: 590_Systole\nMiss: 597_Diastole\nMiss: 597_Systole
\n\nwhen I run Train.py. Anybody has similar one or know why this happen?
\n\nThank you in advance.
\n\n[/quote]\nYes, if you look at the comment above doHist() in Train.py, 2 person missing due to frame selection. The frame selection was done in the preprocessing. The code use another method to generate result for the 2, or you can use your own to handle this problem.
", "rawMarkdown": "[quote=kagglekaggle;105052]\r\n\r\nHello all,\r\n\r\nI am a new B here and simulating this e2e tutorial. So far everything goes fine except for \r\nMiss: 590_Diastole\r\nMiss: 590_Systole\r\nMiss: 597_Diastole\r\nMiss: 597_Systole\r\n\r\nwhen I run Train.py. Anybody has similar one or know why this happen?\r\n\r\nThank you in advance.\r\n\r\n[/quote]\r\nYes, if you look at the comment above doHist() in Train.py, 2 person missing due to frame selection. The frame selection was done in the preprocessing. The code use another method to generate result for the 2, or you can use your own to handle this problem.\r\n" }, { "id": 105052, "postDate": "2016-01-19T13:43:33.473Z", "content": "Hello all,
\n\nI am a new B here and simulating this e2e tutorial. So far everything goes fine except for \nMiss: 590_Diastole\nMiss: 590_Systole\nMiss: 597_Diastole\nMiss: 597_Systole
\n\nwhen I run Train.py. Anybody has similar one or know why this happen?
\n\nThank you in advance.
", "rawMarkdown": "Hello all,\r\n\r\nI am a new B here and simulating this e2e tutorial. So far everything goes fine except for \r\nMiss: 590_Diastole\r\nMiss: 590_Systole\r\nMiss: 597_Diastole\r\nMiss: 597_Systole\r\n\r\nwhen I run Train.py. Anybody has similar one or know why this happen?\r\n\r\nThank you in advance." }, { "id": 105022, "postDate": "2016-01-18T22:55:30.620Z", "content": "[quote=Franc Bračun;104978]
\n\n@Jon Carlies: This would be necessary if you use a library released before the second part of the december.
\n\n[/quote]
\n\nI ended up using Visual Studio 2015 to compile the mxnet .NET Test app that comes with the precompiled mxnet. Visual Studio prompted me to download some other dependencies which I did and now everything is working smoothly. maybe I'll go back later and figure out the details but for now I'm happy to move on with Train.py.
", "rawMarkdown": "[quote=Franc Bračun;104978]\r\n\r\n@Jon Carlies: This would be necessary if you use a library released before the second part of the december.\r\n\r\n[/quote]\r\n\r\nI ended up using Visual Studio 2015 to compile the mxnet .NET Test app that comes with the precompiled mxnet. Visual Studio prompted me to download some other dependencies which I did and now everything is working smoothly. maybe I'll go back later and figure out the details but for now I'm happy to move on with Train.py." }, { "id": 105016, "postDate": "2016-01-18T21:31:32.240Z", "content": "Have you setup graphviz with conda install instead of the system one? Visualizing MXnet is not so hard, see this http://josephpcohen.com/w/visualizing-cnn-architectures-side-by-side-with-mxnet/ \n[quote=WD;105013]
\n\ni am trying to visualize the network. Some others might be interested in this as well. I thought the code below should do the trick. However, even doing all the sudo apt-get installs (ubuntu) i still get a graphviz library not found error. let me know if anyone has experience with this or has encountered the same issue or knows how to resolve this!
\n\n\n\n\nimport graphviz import find_mxnet import mxnet as mx import importlib
\n \ndef get_lenet():\n #the code from the train.py file
\n \nnetwork = get_lenet()
\n \nmx.viz.plot_network(network)
\n
[/quote]
", "rawMarkdown": "Have you setup graphviz with conda install instead of the system one? Visualizing MXnet is not so hard, see this http://josephpcohen.com/w/visualizing-cnn-architectures-side-by-side-with-mxnet/ \r\n[quote=WD;105013]\r\n\r\ni am trying to visualize the network. Some others might be interested in this as well. I thought the code below should do the trick. However, even doing all the sudo apt-get installs (ubuntu) i still get a graphviz library not found error. let me know if anyone has experience with this or has encountered the same issue or knows how to resolve this! \r\n\r\n\r\n> import graphviz import find_mxnet import mxnet as mx import importlib\r\n> \r\n> def get_lenet():\r\n> #the code from the train.py file \r\n\r\n> network = get_lenet()\r\n> \r\n> mx.viz.plot_network(network)\r\n\r\n\r\n\r\n[/quote]\r\n" }, { "id": 104986, "postDate": "2016-01-18T16:47:40.843Z", "content": "@lancerts. wd is weight decay. wd and momentum are both parameters that influence the type of gradient descent. there is a good no-learn tutoral (face point recognition) that has some nifty graphics on the different types of gradients. let me know if this helps
", "rawMarkdown": "@lancerts. wd is weight decay. wd and momentum are both parameters that influence the type of gradient descent. there is a good no-learn tutoral (face point recognition) that has some nifty graphics on the different types of gradients. let me know if this helps" }, { "id": 104981, "postDate": "2016-01-18T15:18:59.287Z", "content": "Newbee here. Can some one explain what is meaning of the momentum and wd parameters in the model? Look up the mxnet document but couldn't find it.
", "rawMarkdown": "Newbee here. Can some one explain what is meaning of the momentum and wd parameters in the model? Look up the mxnet document but couldn't find it." }, { "id": 104978, "postDate": "2016-01-18T15:00:19.440Z", "content": "@Jon Carlies: This would be necessary if you use a library released before the second part of the december.
", "rawMarkdown": "@Jon Carlies: This would be necessary if you use a library released before the second part of the december." }, { "id": 104973, "postDate": "2016-01-18T14:12:09.157Z", "content": "[quote=Franc Bračun;104955]
\n\n@Jon Carlies: since precompiled mxnet libraries are only cpu enabled, replace all : devs = [mx.gpu(0)] with devs = [mx.cpu()].
[/quote]
\n\n@Franc I I have not made it to those lines of code yet. I'm not able to get past the first line, network = get_lenet(). Maybe I have to update the Python code for mxnet?
", "rawMarkdown": "[quote=Franc Bračun;104955]\r\n\r\n@Jon Carlies: since precompiled mxnet libraries are only cpu enabled, replace all : `devs = [mx.gpu(0)]` with `devs = [mx.cpu()]`. \r\n\r\n[/quote]\r\n\r\n@Franc I I have not made it to those lines of code yet. I'm not able to get past the first line, network = get_lenet(). Maybe I have to update the Python code for mxnet?" }, { "id": 104936, "postDate": "2016-01-18T03:42:51.157Z", "content": "[quote=Franc Bračun;103479]
\n\n[quote=Tianqi Chen;103405]
\n\n[quote=Franc Bračun;103338]
\n\n@Mathurin:
\n\nI had the same problem on Windows. I fixed it by adding one line of code root=root.replace('\\\\','/') and it works for me. See the relevant part of code below. Hope this will help you.
for root, _, files in os.walk(root_path):\n root=root.replace('\\\\','/')\n if len(files) == 0 or not files[0].endswith(".dcm") or root.find("sax") == -1:\n continue\n prefix = files[0].rsplit('-', 1)[0]\n fileset = set(files)\n expected = ["%s-%04d.dcm" % (prefix, i + 1) for i in range(30)]\n if all(x in fileset for x in expected):\n ret.append([root + "/" + x for x in expected])\n # sort for reproduciblity\n return sorted(ret, key = lambda x: x[0])\n[/quote]
\n\nThis seems to be a great solution, maybe you can send a PR back so the future windows users can benefit from it, we can use python's platform check to check platforms
\n\n[/quote]
\n\n@ Tianqi Chen: I'm not sure what do you mean by PR. Therefore I have attached "Preprocessing.py" file that I have modified to work properly on windows.
[/quote]
\n\n@Franc Thank you for sharing he preprocessing.py updates. I added code to split_csv to in order to ignore the .jpg files but now am struggling with Train.py. Were you able to get this to work on Windows? I have the precompiled mxnet libraries installed and they do not seem to be working. Curious to know what you did.
", "rawMarkdown": "[quote=Franc Bračun;103479]\r\n\r\n[quote=Tianqi Chen;103405]\r\n\r\n[quote=Franc Bračun;103338]\r\n\r\n@Mathurin:\r\n\r\nI had the same problem on Windows. I fixed it by adding one line of code `root=root.replace('\\\\','/')` and it works for me. See the relevant part of code below. Hope this will help you.\r\n\r\n for root, _, files in os.walk(root_path):\r\n root=root.replace('\\\\','/')\r\n if len(files) == 0 or not files[0].endswith(\".dcm\") or root.find(\"sax\") == -1:\r\n continue\r\n prefix = files[0].rsplit('-', 1)[0]\r\n fileset = set(files)\r\n expected = [\"%s-%04d.dcm\" % (prefix, i + 1) for i in range(30)]\r\n if all(x in fileset for x in expected):\r\n ret.append([root + \"/\" + x for x in expected])\r\n # sort for reproduciblity\r\n return sorted(ret, key = lambda x: x[0])\r\n\r\n[/quote]\r\n\r\nThis seems to be a great solution, maybe you can send a PR back so the future windows users can benefit from it, we can use python's platform check to check platforms\r\n\r\n[/quote]\r\n\r\n@ Tianqi Chen: I'm not sure what do you mean by PR. Therefore I have attached `\"Preprocessing.py\"` file that I have modified to work properly on windows.\r\n\r\n\r\n[/quote]\r\n\r\n@Franc Thank you for sharing he preprocessing.py updates. I added code to split_csv to in order to ignore the .jpg files but now am struggling with Train.py. Were you able to get this to work on Windows? I have the precompiled mxnet libraries installed and they do not seem to be working. Curious to know what you did.\r\n" }, { "id": 104933, "postDate": "2016-01-18T03:32:23.480Z", "content": "[quote=Özgün Genç;102913]
\n\nYou need to reinstall the python package as well.
\n\ncd python; sudo python setup.py install\n[/quote]
\n\nFinally made it to Train.py and immediately hit a stopper with mxnet. I am using the precompiled mxnet library on Windows. I have installed the python package and have set the environment paths but still does not seem to recognize the attributes. \nAttributeError: module 'mxnet' has no attribute 'nd'
Any ideas? Thanks!
", "rawMarkdown": "[quote=Özgün Genç;102913]\r\n\r\nYou need to reinstall the python package as well. \r\n\r\n cd python; sudo python setup.py install\r\n\r\n[/quote]\r\n\r\nFinally made it to Train.py and immediately hit a stopper with mxnet. I am using the precompiled mxnet library on Windows. I have installed the python package and have set the environment paths but still does not seem to recognize the attributes. \r\n`AttributeError: module 'mxnet' has no attribute 'nd'`\r\n\r\nAny ideas? Thanks!" }, { "id": 104930, "postDate": "2016-01-18T03:04:44.057Z", "content": "[quote=Scott Smith;104906]
\n\n@Jon: I think you should have the same number of rows. When I was running into this, I just opened the csv's in notepad++ and visually inspected. You should not have an empty row at the end of any of the csv's, double entries (e.g. in train-label) or empty rows elsewhere. I think the number of rows for train-label and the training vectors should match.
\n\nThe approach I ended up taking though was to just get rid of Windows and move my laptop to Ubuntu 15.04 + anaconda3 + pydicom + CUDA 7.5 + CUDNN 3 + openblas + opencv + mxnet. Having no experience with linux, being a mechanical engineer and really only focusing on R for this late-night hobby, it took a solid week to get through.
\n\n[/quote]
\n\n@Scott Smith\nThanks for leading me down the right path. I was able to resolve this. Again, the .jpg files are the root cause of the index issue. That is, def split_csv does not seem to account for the .jpg files and will create empty rows in "./train-64x64-data.csv". By ignoring these empty rows, I am able to get 5293 rows which matches my train_index.
\n\nHere's my split_csv function now:
\n\n def split_csv(src_csv, split_to_train, train_csv, test_csv):\n ftrain = open(train_csv, "w")\n ftest = open(test_csv, "w")\n cnt = 0\n for l in open(src_csv):\n **if not l.strip():**\n if split_to_train[cnt]:\n ftrain.write(l)\n else:\n ftest.write(l)\n cnt = cnt + 1\n print(cnt)\n ftrain.close()\n ftest.close()\n[quote=Scott Smith;104906]
\n\n@Jon: I think you should have the same number of rows. When I was running into this, I just opened the csv's in notepad++ and visually inspected. You should not have an empty row at the end of any of the csv's, double entries (e.g. in train-label) or empty rows elsewhere. I think the number of rows for train-label and the training vectors should match.
\n\nThe approach I ended up taking though was to just get rid of Windows and move my laptop to Ubuntu 15.04 + anaconda3 + pydicom + CUDA 7.5 + CUDNN 3 + openblas + opencv + mxnet. Having no experience with linux, being a mechanical engineer and really only focusing on R for this late-night hobby, it took a solid week to get through.
\n\n[/quote]
\n\n@Scott Smith\nThanks for leading me down the right path. I was able to resolve this. Again, the .jpg files are the root cause of the index issue. That is, def split_csv does not seem to account for the .jpg files and will create empty rows in "./train-64x64-data.csv". By ignoring these empty rows, I am able to get 5293 rows which matches my train_index.
\n\nHere's my split_csv function now:
\n\ndef split_csv(src_csv, split_to_train, train_csv, test_csv):\nftrain = open(train_csv, "w")\n ftest = open(test_csv, "w")\n cnt = 0\n for l in open(src_csv):\n if not l.strip():\n if split_to_train[cnt]:\n ftrain.write(l)\n else:\n ftest.write(l)\n cnt = cnt + 1\n print(cnt)\n ftrain.close()\n ftest.close()
", "rawMarkdown": "[quote=Scott Smith;104906]\r\n\r\n@Jon: I think you should have the same number of rows. When I was running into this, I just opened the csv's in notepad++ and visually inspected. You should not have an empty row at the end of any of the csv's, double entries (e.g. in train-label) or empty rows elsewhere. I think the number of rows for train-label and the training vectors should match.\r\n\r\nThe approach I ended up taking though was to just get rid of Windows and move my laptop to Ubuntu 15.04 + anaconda3 + pydicom + CUDA 7.5 + CUDNN 3 + openblas + opencv + mxnet. Having no experience with linux, being a mechanical engineer and really only focusing on R for this late-night hobby, it took a solid week to get through.\r\n\r\n[/quote]\r\n\r\n @Scott Smith\r\nThanks for leading me down the right path. I was able to resolve this. Again, the .jpg files are the root cause of the index issue. That is, def split_csv does not seem to account for the .jpg files and will create empty rows in \"./train-64x64-data.csv\". By ignoring these empty rows, I am able to get 5293 rows which matches my train_index.\r\n\r\nHere's my split_csv function now:\r\n\r\n def split_csv(src_csv, split_to_train, train_csv, test_csv):\r\n ftrain = open(train_csv, \"w\")\r\n ftest = open(test_csv, \"w\")\r\n cnt = 0\r\n for l in open(src_csv):\r\n **if not l.strip():**\r\n if split_to_train[cnt]:\r\n ftrain.write(l)\r\n else:\r\n ftest.write(l)\r\n cnt = cnt + 1\r\n print(cnt)\r\n ftrain.close()\r\n ftest.close()" }, { "id": 104906, "postDate": "2016-01-17T20:38:43.173Z", "content": "@Jon: I think you should have the same number of rows. When I was running into this, I just opened the csv's in notepad++ and visually inspected. You should not have an empty row at the end of any of the csv's, double entries (e.g. in train-label) or empty rows elsewhere. I think the number of rows for train-label and the training vectors should match.
\n\nThe approach I ended up taking though was to just get rid of Windows and move my laptop to Ubuntu 15.04 + anaconda3 + pydicom + CUDA 7.5 + CUDNN 3 + openblas + opencv + mxnet. Having no experience with linux, being a mechanical engineer and really only focusing on R for this late-night hobby, it took a solid week to get through.
", "rawMarkdown": "@Jon: I think you should have the same number of rows. When I was running into this, I just opened the csv's in notepad++ and visually inspected. You should not have an empty row at the end of any of the csv's, double entries (e.g. in train-label) or empty rows elsewhere. I think the number of rows for train-label and the training vectors should match.\r\n\r\nThe approach I ended up taking though was to just get rid of Windows and move my laptop to Ubuntu 15.04 + anaconda3 + pydicom + CUDA 7.5 + CUDNN 3 + openblas + opencv + mxnet. Having no experience with linux, being a mechanical engineer and really only focusing on R for this late-night hobby, it took a solid week to get through." }, { "id": 104883, "postDate": "2016-01-17T18:18:08.407Z", "content": "@EIGSI - many thanks for sharing this. i had overlooked especially the last line in that code. On another topic - how did you create the validation curves that you posted earlier? Did you use the code on https://github.com/dmlc/mxnet/issues/511 (which i am still trying to get to work) or did you take another approach?
", "rawMarkdown": "@EIGSI - many thanks for sharing this. i had overlooked especially the last line in that code. On another topic - how did you create the validation curves that you posted earlier? Did you use the code on https://github.com/dmlc/mxnet/issues/511 (which i am still trying to get to work) or did you take another approach? " }, { "id": 104867, "postDate": "2016-01-17T15:17:18.793Z", "content": "this code in train.py does the averaging:
\n\ndef accumulate_result(validate_lst, prob):\n sum_result = {}\n cnt_result = {}\n size = prob.shape[0]\n fi = csv.reader(open(validate_lst))\n for i in range(size):\n line = fi.__next__() # Python2: line = fi.next()\n idx = int(line[0])\n if idx not in cnt_result:\n cnt_result[idx] = 0.\n sum_result[idx] = np.zeros((1, prob.shape[1]))\n cnt_result[idx] += 1\n sum_result[idx] += prob[i, :]\n for i in cnt_result.keys():\n sum_result[i][:] /= cnt_result[i]\n return sum_result\n[quote=WD;104850]
\n\nFrom this point my understanding gets hazy. I dont fully understand how we move from the output from the prediction (which has 1048 rows or 1048-30-image-stacks) to the final submission (where we have one row for each of the 200 patients). I would image that we have to do some averaging between different predictions for different stacks for a given patient - but i dont see this in the code. Any help much appreciated
\n\n[/quote]
", "rawMarkdown": "this code in train.py does the averaging:\r\n\r\n def accumulate_result(validate_lst, prob):\r\n sum_result = {}\r\n cnt_result = {}\r\n size = prob.shape[0]\r\n fi = csv.reader(open(validate_lst))\r\n for i in range(size):\r\n line = fi.__next__() # Python2: line = fi.next()\r\n idx = int(line[0])\r\n if idx not in cnt_result:\r\n cnt_result[idx] = 0.\r\n sum_result[idx] = np.zeros((1, prob.shape[1]))\r\n cnt_result[idx] += 1\r\n sum_result[idx] += prob[i, :]\r\n for i in cnt_result.keys():\r\n sum_result[i][:] /= cnt_result[i]\r\n return sum_result\r\n\r\n[quote=WD;104850]\r\n\r\n\r\nFrom this point my understanding gets hazy. I dont fully understand how we move from the output from the prediction (which has 1048 rows or 1048-30-image-stacks) to the final submission (where we have one row for each of the 200 patients). I would image that we have to do some averaging between different predictions for different stacks for a given patient - but i dont see this in the code. Any help much appreciated\r\n\r\n\r\n\r\n[/quote]\r\n" }, { "id": 104827, "postDate": "2016-01-17T02:21:49.657Z", "content": "For each breath-hold sequence the data contains 30 frames. The first half or so is systole and the rest of the frames represent diastole. Considering this, I tried to divide the data into two and fed to the respective networks. Although processing time drops significantly, I did not seem to gain much in terms of accuracy which was somewhat disappointing
", "rawMarkdown": "For each breath-hold sequence the data contains 30 frames. The first half or so is systole and the rest of the frames represent diastole. Considering this, I tried to divide the data into two and fed to the respective networks. Although processing time drops significantly, I did not seem to gain much in terms of accuracy which was somewhat disappointing" }, { "id": 104744, "postDate": "2016-01-16T05:57:11.787Z", "content": "After implementing the root=root.replace('\\','/') I get a blank data frame returned from get_frames.
\nThis is the second time running through the code. I think it might have something to do with the .jpg files introduced as noted in another post. Will try to code around them and see if that does the trick.
Here is my code:
\n\n def get_frames(root_path):\n """Get path to all the frame in view SAX and contain complete frames"""\n print('get_frames')\n ret = []\n for root, _, files in os.walk(root_path):\n root=root.replace('\\\\','/')\n #print(files)\n if len(files) == 0 or not files[0].endswith(".dcm") or root.find("sax") == -1:\n continue\n prefix = files[0].rsplit('-', 1)[0]\n fileset = set(files)\n expected = ["%s-%04d.dcm" % (prefix, i + 1) for i in range(30)]\n if all(x in fileset for x in expected):\n ret.append([root + "/" + x for x in expected])\n # sort for reproduciblity\n return sorted(ret, key = lambda x: x[0])\n\ntrain_frames = get_frames("./data/train")\nAny help would be greatly appreciated.
\n\nThank you.
\n\nUPDATE: This post fixed my issue. Thank you!
", "rawMarkdown": "After implementing the root=root.replace('\\\\','/') I get a blank data frame returned from get_frames. \r\nThis is the second time running through the code. I think it might have something to do with the .jpg files introduced as noted in another post. Will try to code around them and see if that does the trick.\r\n\r\nHere is my code:\r\n\r\n \r\n\r\n def get_frames(root_path):\r\n \"\"\"Get path to all the frame in view SAX and contain complete frames\"\"\"\r\n print('get_frames')\r\n ret = []\r\n for root, _, files in os.walk(root_path):\r\n root=root.replace('\\\\','/')\r\n #print(files)\r\n if len(files) == 0 or not files[0].endswith(\".dcm\") or root.find(\"sax\") == -1:\r\n continue\r\n prefix = files[0].rsplit('-', 1)[0]\r\n fileset = set(files)\r\n expected = [\"%s-%04d.dcm\" % (prefix, i + 1) for i in range(30)]\r\n if all(x in fileset for x in expected):\r\n ret.append([root + \"/\" + x for x in expected])\r\n # sort for reproduciblity\r\n return sorted(ret, key = lambda x: x[0])\r\n \r\n train_frames = get_frames(\"./data/train\")\r\n\r\nAny help would be greatly appreciated.\r\n\r\nThank you.\r\n\r\nUPDATE: [This post fixed my issue. Thank you!][1]\r\n\r\n\r\n [1]: https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/18079/end-to-end-deep-learning-tutorial-0-0392/103854#post103854" }, { "id": 104743, "postDate": "2016-01-16T05:50:29.627Z", "content": "[quote=Franc Bračun;103479]
\n\n[quote=Tianqi Chen;103405]
\n\n[quote=Franc Bračun;103338]
\n\n@Mathurin:
\n\nI had the same problem on Windows. I fixed it by adding one line of code root=root.replace('\\\\','/') and it works for me. See the relevant part of code below. Hope this will help you.
for root, _, files in os.walk(root_path):\n root=root.replace('\\\\','/')\n if len(files) == 0 or not files[0].endswith(".dcm") or root.find("sax") == -1:\n continue\n prefix = files[0].rsplit('-', 1)[0]\n fileset = set(files)\n expected = ["%s-%04d.dcm" % (prefix, i + 1) for i in range(30)]\n if all(x in fileset for x in expected):\n ret.append([root + "/" + x for x in expected])\n # sort for reproduciblity\n return sorted(ret, key = lambda x: x[0])\n[/quote]
\n\nThis seems to be a great solution, maybe you can send a PR back so the future windows users can benefit from it, we can use python's platform check to check platforms
\n\n[/quote]
\n\n@ Tianqi Chen: I'm not sure what do you mean by PR. Therefore I have attached "Preprocessing.py" file that I have modified to work properly on windows.
[/quote]
", "rawMarkdown": "[quote=Franc Bračun;103479]\r\n\r\n[quote=Tianqi Chen;103405]\r\n\r\n[quote=Franc Bračun;103338]\r\n\r\n@Mathurin:\r\n\r\nI had the same problem on Windows. I fixed it by adding one line of code `root=root.replace('\\\\','/')` and it works for me. See the relevant part of code below. Hope this will help you.\r\n\r\n for root, _, files in os.walk(root_path):\r\n root=root.replace('\\\\','/')\r\n if len(files) == 0 or not files[0].endswith(\".dcm\") or root.find(\"sax\") == -1:\r\n continue\r\n prefix = files[0].rsplit('-', 1)[0]\r\n fileset = set(files)\r\n expected = [\"%s-%04d.dcm\" % (prefix, i + 1) for i in range(30)]\r\n if all(x in fileset for x in expected):\r\n ret.append([root + \"/\" + x for x in expected])\r\n # sort for reproduciblity\r\n return sorted(ret, key = lambda x: x[0])\r\n\r\n[/quote]\r\n\r\nThis seems to be a great solution, maybe you can send a PR back so the future windows users can benefit from it, we can use python's platform check to check platforms\r\n\r\n[/quote]\r\n\r\n@ Tianqi Chen: I'm not sure what do you mean by PR. Therefore I have attached `\"Preprocessing.py\"` file that I have modified to work properly on windows.\r\n\r\n\r\n[/quote]\r\n" }, { "id": 104428, "postDate": "2016-01-12T16:42:35.197Z", "content": "In order to diagnose the problem, i made the files slightly smaller by changing the line in the Preprocessing file to
\n\nif len(files) == 0 or not files[0].endswith(".dcm") or root.find("sax_9") == -1:
\n\nTHis reduces the size of the csv files created - but still - the Train.Py does not work (per above)
\n\nI think that the GPU is not working properly, although i used a pre-installed AMS instance that has CUDA installed. Will reserach more
", "rawMarkdown": "In order to diagnose the problem, i made the files slightly smaller by changing the line in the Preprocessing file to \r\n\r\nif len(files) == 0 or not files[0].endswith(\".dcm\") or root.find(\"sax_9\") == -1:\r\n\r\nTHis reduces the size of the csv files created - but still - the Train.Py does not work (per above)\r\n\r\nI think that the GPU is not working properly, although i used a pre-installed AMS instance that has CUDA installed. Will reserach more \r\n\r\n" }, { "id": 104352, "postDate": "2016-01-12T02:02:02.967Z", "content": "I am encountering the following Mxnet problem. After running python Train.py I am just seeing
\n\nINFO:root:Start training with [gpu(0)]
\n\nand subsequently no furhter information, and no results or outputs. I tried to write stderr and stdout to file as well - but i received no other information. The system just runs and then crashes.
\n\nI am using the following setup (please let me know if i am doing something wrong / missing something):
\n\nsudo apt-get update\nsudo apt-get install -y build-essential git libcurl4-openssl-dev libatlas-base-dev libopencv-dev python-numpy\nsudo apt-get -y install python-numpy python-scipy python-matplotlib \nsudo apt-get -y install python-dicom python-skimage
\n\ngit clone --recursive https://github.com/dmlc/mxnet\ncd mxnet; cp make/config.mk .\necho "USE_CUDA=1" >>config.mk\necho "USE_CUDA_PATH=/usr/local/cuda" >>config.mk\necho "USE_CUDNN=1" >>config.mk\necho "USE_BLAS=atlas" >> config.mk\necho "USE_DIST_KVSTORE = 1" >>config.mk\necho "USE_S3=1" >>config.mk\nmake -j8
\n\nlet me know any tips / help!
", "rawMarkdown": "I am encountering the following Mxnet problem. After running python Train.py I am just seeing \r\n\r\nINFO:root:Start training with [gpu(0)] \r\n\r\nand subsequently no furhter information, and no results or outputs. I tried to write stderr and stdout to file as well - but i received no other information. The system just runs and then crashes. \r\n\r\nI am using the following setup (please let me know if i am doing something wrong / missing something):\r\n\r\n* I am using an AWS EC2 G2.2x large CPU-enabled AMI that already has cuDnn and cuda installed, and i download the training data on a seperate EBS volume following the specified file structure\r\n* I then run the following code:\r\n\r\nsudo apt-get update\r\nsudo apt-get install -y build-essential git libcurl4-openssl-dev libatlas-base-dev libopencv-dev python-numpy\r\nsudo apt-get -y install python-numpy python-scipy python-matplotlib \r\nsudo apt-get -y install python-dicom python-skimage \r\n\r\n* And i install MxNet with the following setup:\r\n\r\ngit clone --recursive https://github.com/dmlc/mxnet\r\ncd mxnet; cp make/config.mk .\r\necho \"USE_CUDA=1\" >>config.mk\r\necho \"USE_CUDA_PATH=/usr/local/cuda\" >>config.mk\r\necho \"USE_CUDNN=1\" >>config.mk\r\necho \"USE_BLAS=atlas\" >> config.mk\r\necho \"USE_DIST_KVSTORE = 1\" >>config.mk\r\necho \"USE_S3=1\" >>config.mk\r\nmake -j8\r\n\r\n* Then i run Preprocessing.py, and subsequetly the Train.py file\r\n\r\nlet me know any tips / help! \r\n\r\n" }, { "id": 104248, "postDate": "2016-01-10T23:48:37.280Z", "content": "The [0] refers to to a specific GPU; e.g. if your machine had two GPUs, gpu[0] would refer to the first and gpu[1] to the second.
", "rawMarkdown": "The [0] refers to to a specific GPU; e.g. if your machine had two GPUs, gpu[0] would refer to the first and gpu[1] to the second." }, { "id": 104246, "postDate": "2016-01-10T23:22:39.373Z", "content": "@patruff - does [gpu(0)] mean that MxNEt is using the GPU - or does the 0 mean that it is not enabled - and hence is running on CPU?
", "rawMarkdown": "@patruff - does [gpu(0)] mean that MxNEt is using the GPU - or does the 0 mean that it is not enabled - and hence is running on CPU? " }, { "id": 104242, "postDate": "2016-01-10T22:05:24.943Z", "content": "@patruff - on an AWS G2.2large - any indication on how long the training might take? There do not seem to be too many indications on overall duration while the algorithim is running (e.g. % of epochs, etc.)
", "rawMarkdown": "@patruff - on an AWS G2.2large - any indication on how long the training might take? There do not seem to be too many indications on overall duration while the algorithim is running (e.g. % of epochs, etc.) " }, { "id": 104241, "postDate": "2016-01-10T22:03:21.350Z", "content": "@waheguru - i used wget - like per below- but on an EBS instance, on the command line:\nwget --load-cookies cookies.txt -nH https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.csv.zip.
\n\nlet me know if u need more help
", "rawMarkdown": "@waheguru - i used wget - like per below- but on an EBS instance, on the command line:\r\nwget --load-cookies cookies.txt -nH https://www.kaggle.com/c/second-annual-data-science-bowl/download/train.csv.zip. \r\n\r\nlet me know if u need more help\r\n" }, { "id": 104229, "postDate": "2016-01-10T18:12:09.043Z", "content": "For those of you using AWS S3/EC2, what's the best (and fastest) way to get the 48 gigs of data onto S3?
", "rawMarkdown": "For those of you using AWS S3/EC2, what's the best (and fastest) way to get the 48 gigs of data onto S3?" }, { "id": 104228, "postDate": "2016-01-10T18:11:44.787Z", "content": "For those of you using AWS S3/EC2, what's the best (and fastest) way to get the 48 gigs of data onto S3?
", "rawMarkdown": "For those of you using AWS S3/EC2, what's the best (and fastest) way to get the 48 gigs of data onto S3?" }, { "id": 104227, "postDate": "2016-01-10T18:04:03.040Z", "content": "@WD, that means it's working, it may take a while to complete.
", "rawMarkdown": "@WD, that means it's working, it may take a while to complete." }, { "id": 104226, "postDate": "2016-01-10T17:52:47.180Z", "content": "Okay, so I figured out what I was doing wrong, python3 did not have mxnet but python2 did. So after installing CUDA, editing the config file, and updating all the modules in python3 everything is working fine. Thanks again for the tutorial.
", "rawMarkdown": " Okay, so I figured out what I was doing wrong, python3 did not have mxnet but python2 did. So after installing CUDA, editing the config file, and updating all the modules in python3 everything is working fine. Thanks again for the tutorial." }, { "id": 104223, "postDate": "2016-01-10T17:15:00.083Z", "content": "Hi guys,
\n\nI took the latest version of Preprocessing.py and suddenly getting a weird error:\nmxnet-master\\dmlc-core\\include\\dmlc./logging.h:208: mxnet-master\\src\\io\\iter_csv.cc:105: Check failed: (row.length) == (shape.Size()) The data size in CSV do not match size of shape: specified shape=(30, 64, 64), the csv row-length=4096.
\nAny thoughts???
I have attached a sample of the input files used in Train.py, when getting the above error. Would really appreciate if someone can please provide samples from what they use to call the training script...maybe that can help me in understanding where's the error.
\n\nMany thanks!
", "rawMarkdown": "Hi guys,\r\n\r\nI took the latest version of Preprocessing.py and suddenly getting a weird error:\r\nmxnet-master\\dmlc-core\\include\\dmlc\\./logging.h:208: mxnet-master\\src\\io\\iter_csv.cc:105: Check failed: (row.length) == (shape.Size()) The data size in CSV do not match size of shape: specified shape=(30, 64, 64), the csv row-length=4096. \r\nAny thoughts???\r\n\r\nI have attached a sample of the input files used in Train.py, when getting the above error. Would really appreciate if someone can please provide samples from what they use to call the training script...maybe that can help me in understanding where's the error.\r\n\r\nMany thanks!" }, { "id": 104222, "postDate": "2016-01-10T17:11:12.097Z", "content": "maybe a follow-up question - my computer has been stuck for a while now on the following status. is this just a sign that the computer is crunching in the background or has the problem hit an error?
\n\nINFO:root:Start training with [gpu(0)]
\n\nWouter
", "rawMarkdown": "maybe a follow-up question - my computer has been stuck for a while now on the following status. is this just a sign that the computer is crunching in the background or has the problem hit an error?\r\n\r\nINFO:root:Start training with [gpu(0)]\r\n\r\nWouter " }, { "id": 104218, "postDate": "2016-01-10T15:46:46.027Z", "content": "this is a great tutoral.
\n\nI am trying to run the tutoral - and have two questions:
\n\nMxNet looks very exciting!
\n\nW
", "rawMarkdown": "this is a great tutoral. \r\n\r\nI am trying to run the tutoral - and have two questions:\r\n\r\n - I am getting a failed to initialize libdc1394 error. What does that mean? It seemingly doesnt stop MxNet from running the Mnist example. How important is this error - and how does one resolve it?\r\n - My instance (on AWS) says INFO root: Start training with [gpu(0)]. Does that mean that GPU is enabled? or not enabled? \r\n\r\nMxNet looks very exciting!\r\n\r\nW" }, { "id": 104166, "postDate": "2016-01-09T21:17:26.530Z", "content": "[quote=earino;104164]
\n\n[quote=Matthew Tubs;104163]\nFranc
\n\nI've been trying to get the Preprocessing.py file you attached running and I've ran into a issue. Would you help me out in understanding what is going on? I am running Python 3.5 on windows, and I am receiving the following error.
\n\ntran_index = np.loadtxt("./tran_label.csv", delimiter=",")[:,0].astype("int")
\n\nIndexError: too many indices for array
\n\nThanks much\n [/quote]
\n\nis that tran_label supposed to be train_label?
\n\n[/quote]\nI did a replace all to see if the problem was something attached to the name of the file.
", "rawMarkdown": "[quote=earino;104164]\r\n\r\n[quote=Matthew Tubs;104163]\r\nFranc \r\n\r\nI've been trying to get the Preprocessing.py file you attached running and I've ran into a issue. Would you help me out in understanding what is going on? I am running Python 3.5 on windows, and I am receiving the following error.\r\n\r\n tran_index = np.loadtxt(\"./tran_label.csv\", delimiter=\",\")[:,0].astype(\"int\")\r\n\r\nIndexError: too many indices for array\r\n\r\nThanks much\r\n [/quote]\r\n\r\nis that tran_label supposed to be train_label?\r\n\r\n\r\n[/quote]\r\nI did a replace all to see if the problem was something attached to the name of the file." }, { "id": 104164, "postDate": "2016-01-09T21:06:13.190Z", "content": "[quote=Matthew Tubs;104163]\nFranc
\n\nI've been trying to get the Preprocessing.py file you attached running and I've ran into a issue. Would you help me out in understanding what is going on? I am running Python 3.5 on windows, and I am receiving the following error.
\n\ntran_index = np.loadtxt("./tran_label.csv", delimiter=",")[:,0].astype("int")
\n\nIndexError: too many indices for array
\n\nThanks much\n [/quote]
\n\nis that tran_label supposed to be train_label?
", "rawMarkdown": "[quote=Matthew Tubs;104163]\r\nFranc \r\n\r\nI've been trying to get the Preprocessing.py file you attached running and I've ran into a issue. Would you help me out in understanding what is going on? I am running Python 3.5 on windows, and I am receiving the following error.\r\n\r\n tran_index = np.loadtxt(\"./tran_label.csv\", delimiter=\",\")[:,0].astype(\"int\")\r\n\r\nIndexError: too many indices for array\r\n\r\nThanks much\r\n [/quote]\r\n\r\nis that tran_label supposed to be train_label?\r\n" }, { "id": 104163, "postDate": "2016-01-09T21:02:18.677Z", "content": "Franc
\n\nI've been trying to get the Preprocessing.py file you attached running and I've ran into a issue. Would you help me out in understanding what is going on? I am running Python 3.5 on windows, and I am receiving the following error.
\n\ntran_index = np.loadtxt("./tran_label.csv", delimiter=",")[:,0].astype("int")
\n\nIndexError: too many indices for array
\n\nThanks much
", "rawMarkdown": "Franc \r\n\r\nI've been trying to get the Preprocessing.py file you attached running and I've ran into a issue. Would you help me out in understanding what is going on? I am running Python 3.5 on windows, and I am receiving the following error.\r\n\r\n tran_index = np.loadtxt(\"./tran_label.csv\", delimiter=\",\")[:,0].astype(\"int\")\r\n\r\nIndexError: too many indices for array\r\n\r\nThanks much\r\n " }, { "id": 104150, "postDate": "2016-01-09T17:26:45.663Z", "content": "@phunter thanks! I should have looked further up the repo! Gah how embarrassing :) cheers!
", "rawMarkdown": "@phunter thanks! I should have looked further up the repo! Gah how embarrassing :) cheers!" }, { "id": 104148, "postDate": "2016-01-09T17:19:47.970Z", "content": "@earino according to https://github.com/dmlc/mxnet it is :
\n\nLicense
\n\n© Contributors, 2015. Licensed under an Apache-2.0 license.
", "rawMarkdown": "@earino according to https://github.com/dmlc/mxnet it is :\r\n\r\nLicense\r\n\r\n© Contributors, 2015. Licensed under an Apache-2.0 license." }, { "id": 104111, "postDate": "2016-01-09T04:57:09.143Z", "content": "@IAslam, Thanks! This contest has a long way to go though; I'm guessing the winning score will be below 0.01, but we shall see. It will be interesting to see what approaches emerge victorious here. I wouldn't be surprised if most of the top scores are using improved versions of this tutorial at this point, but I suspect that approach won't be enough on it's own.
", "rawMarkdown": "@IAslam, Thanks! This contest has a long way to go though; I'm guessing the winning score will be below 0.01, but we shall see. It will be interesting to see what approaches emerge victorious here. I wouldn't be surprised if most of the top scores are using improved versions of this tutorial at this point, but I suspect that approach won't be enough on it's own. \r\n" }, { "id": 104110, "postDate": "2016-01-09T04:51:44.557Z", "content": "I got some strange mxnet behaviour and am wondering if I may have a bug somewhere or this is just non-convergence.
\n\nMy systole results were as expected. For diastole data I first got predictions that were only zero or one for each probability column (eg for slice 1: 0,0,...,1,0,1,1,0,0,1,0,1,1,...,1). These were different for each patient (edit: slice, not patient) and roughly did correspond to the data (as far as such a result can). Then I increased the learning rate and got values intermediate between zero and one, but the exact same curve for every patient (edit: slice, not patient) .
\n\nDoes this sounds like something mxnet would normally produce, or would you expect a bug? I was using the default get.lenet() function and prebuilt R package with cpu.
", "rawMarkdown": "I got some strange mxnet behaviour and am wondering if I may have a bug somewhere or this is just non-convergence. \r\n\r\nMy systole results were as expected. For diastole data I first got predictions that were only zero or one for each probability column (eg for slice 1: 0,0,...,1,0,1,1,0,0,1,0,1,1,...,1). These were different for each patient (**edit:** slice, not patient) and roughly did correspond to the data (as far as such a result can). Then I increased the learning rate and got values intermediate between zero and one, but the exact same curve for every patient (**edit:** slice, not patient) . \r\n\r\nDoes this sounds like something mxnet would normally produce, or would you expect a bug? I was using the default get.lenet() function and prebuilt R package with cpu." }, { "id": 104090, "postDate": "2016-01-09T00:52:42.010Z", "content": "@Tim Hochberg, My interpretation of Shannon's response is the same. I have asked a clarification from Shannon regarding the segmentation output 'requirement' on the other thread, just to be sure. We don't need surprises too far along the competition. And, congratulations on your leaderboard score. Way to go!
", "rawMarkdown": "@Tim Hochberg, My interpretation of Shannon's response is the same. I have asked a clarification from Shannon regarding the segmentation output 'requirement' on the other thread, just to be sure. We don't need surprises too far along the competition. And, congratulations on your leaderboard score. Way to go!" }, { "id": 104064, "postDate": "2016-01-08T21:18:56.637Z", "content": "Thanks for the really good starter code. But, besides the two volumes that need to be estimated, a segmentation output (https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/18198/q-a-with-principle-investigators-michael-hansen-ph-d-and-dr-andrew-arai/104034#post104034) is also expected and I am not sure this approach can produce that.
", "rawMarkdown": "Thanks for the really good starter code. But, besides the two volumes that need to be estimated, a segmentation output (https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/18198/q-a-with-principle-investigators-michael-hansen-ph-d-and-dr-andrew-arai/104034#post104034) is also expected and I am not sure this approach can produce that." }, { "id": 104014, "postDate": "2016-01-08T15:13:19.617Z", "content": "I get an odd error when trying to run python3 Train.py
\n\nFile "Train.py", line 8, in \n import mxnet as mx\n...\nOSError: /usr/local/lib/python3.4/dist-packages/mxnet-0.5.0-py3.4.egg/mxnet/libmxnet.so: undefined symbol: omp_get_thread_num
", "rawMarkdown": "I get an odd error when trying to run python3 Train.py\r\n\r\nFile \"Train.py\", line 8, inThanks for the tutorial, really appreciate it.
", "rawMarkdown": " Thanks for the tutorial, really appreciate it." }, { "id": 103897, "postDate": "2016-01-07T17:09:05.867Z", "content": "Thanks for this tutorial and introduction to mxnet. It's excellent.
", "rawMarkdown": "Thanks for this tutorial and introduction to mxnet. It's excellent." }, { "id": 103801, "postDate": "2016-01-06T16:09:04.820Z", "content": "I have not been able to run the model myself, but am studying the code, and as a beginner, was hoping that I could ask some basic questions:
\n\nPlease correct me if I am wrong,\nAll the best, \nWD
", "rawMarkdown": "I have not been able to run the model myself, but am studying the code, and as a beginner, was hoping that I could ask some basic questions: \r\n\r\n - Is the input of this model a concatenated matrix of all the individual “difference” matrices between subsequent images of a stack? E.g. the matrix would have, if an image is 64*64, a shape of 64*1920?\r\n - Is the output a logistic regression between 600 end-neurons and the dependent variable (the volume labels)?\r\n\r\nPlease correct me if I am wrong,\r\nAll the best, \r\nWD\r\n" }, { "id": 103546, "postDate": "2016-01-04T07:47:24.527Z", "content": "[quote=Franc Bračun;103479]
\n\n[quote=Tianqi Chen;103405]
\n\n[quote=Franc Bračun;103338]
\n\n@Mathurin:
\n\nI had the same problem on Windows. I fixed it by adding one line of code root=root.replace('\\\\','/') and it works for me. See the relevant part of code below. Hope this will help you.
for root, _, files in os.walk(root_path):\n root=root.replace('\\\\','/')\n if len(files) == 0 or not files[0].endswith(".dcm") or root.find("sax") == -1:\n continue\n prefix = files[0].rsplit('-', 1)[0]\n fileset = set(files)\n expected = ["%s-%04d.dcm" % (prefix, i + 1) for i in range(30)]\n if all(x in fileset for x in expected):\n ret.append([root + "/" + x for x in expected])\n # sort for reproduciblity\n return sorted(ret, key = lambda x: x[0])\n[/quote]
\n\nThis seems to be a great solution, maybe you can send a PR back so the future windows users can benefit from it, we can use python's platform check to check platforms
\n\n[/quote]
\n\n@ Tianqi Chen: I'm not sure what do you mean by PR. Therefore I have attached "Preprocessing.py" file that I have modified to work properly on windows.
[/quote]
\n\nI mean https://help.github.com/articles/using-pull-requests/ \nIt would be great if you can do it with your github account so your contribution will be recorded!
", "rawMarkdown": "[quote=Franc Bračun;103479]\r\n\r\n[quote=Tianqi Chen;103405]\r\n\r\n[quote=Franc Bračun;103338]\r\n\r\n@Mathurin:\r\n\r\nI had the same problem on Windows. I fixed it by adding one line of code `root=root.replace('\\\\','/')` and it works for me. See the relevant part of code below. Hope this will help you.\r\n\r\n for root, _, files in os.walk(root_path):\r\n root=root.replace('\\\\','/')\r\n if len(files) == 0 or not files[0].endswith(\".dcm\") or root.find(\"sax\") == -1:\r\n continue\r\n prefix = files[0].rsplit('-', 1)[0]\r\n fileset = set(files)\r\n expected = [\"%s-%04d.dcm\" % (prefix, i + 1) for i in range(30)]\r\n if all(x in fileset for x in expected):\r\n ret.append([root + \"/\" + x for x in expected])\r\n # sort for reproduciblity\r\n return sorted(ret, key = lambda x: x[0])\r\n\r\n[/quote]\r\n\r\nThis seems to be a great solution, maybe you can send a PR back so the future windows users can benefit from it, we can use python's platform check to check platforms\r\n\r\n[/quote]\r\n\r\n@ Tianqi Chen: I'm not sure what do you mean by PR. Therefore I have attached `\"Preprocessing.py\"` file that I have modified to work properly on windows.\r\n\r\n\r\n[/quote]\r\n\r\nI mean https://help.github.com/articles/using-pull-requests/ \r\nIt would be great if you can do it with your github account so your contribution will be recorded!" }, { "id": 103529, "postDate": "2016-01-04T01:21:31.967Z", "content": "The GPU on my mac has 1GB memory. I have been trying to run this example but I am getting cuda out of memory errors on Train.py. I reduced batch size from 32 to 16 and also tried reducing epochs from 65 to 10. It seems I always get the memory error right at the last epoch. Did anyone else experience the same? I installed cuda 7.5 on yosemite
", "rawMarkdown": "The GPU on my mac has 1GB memory. I have been trying to run this example but I am getting cuda out of memory errors on Train.py. I reduced batch size from 32 to 16 and also tried reducing epochs from 65 to 10. It seems I always get the memory error right at the last epoch. Did anyone else experience the same? I installed cuda 7.5 on yosemite" }, { "id": 103479, "postDate": "2016-01-03T15:37:54.727Z", "content": "[quote=Tianqi Chen;103405]
\n\n[quote=Franc Bračun;103338]
\n\n@Mathurin:
\n\nI had the same problem on Windows. I fixed it by adding one line of code root=root.replace('\\\\','/') and it works for me. See the relevant part of code below. Hope this will help you.
for root, _, files in os.walk(root_path):\n root=root.replace('\\\\','/')\n if len(files) == 0 or not files[0].endswith(".dcm") or root.find("sax") == -1:\n continue\n prefix = files[0].rsplit('-', 1)[0]\n fileset = set(files)\n expected = ["%s-%04d.dcm" % (prefix, i + 1) for i in range(30)]\n if all(x in fileset for x in expected):\n ret.append([root + "/" + x for x in expected])\n # sort for reproduciblity\n return sorted(ret, key = lambda x: x[0])\n[/quote]
\n\nThis seems to be a great solution, maybe you can send a PR back so the future windows users can benefit from it, we can use python's platform check to check platforms
\n\n[/quote]
\n\n@ Tianqi Chen: I'm not sure what do you mean by PR. Therefore I have attached "Preprocessing.py" file that I have modified to work properly on windows.
Also, shouldn't encode_label have <= instead of < for the following lines?
\n\nstytole_encode = np.array([\n (x <= np.arange(600)) for x in stytole\n ], dtype=np.uint8)\ndiastole_encode = np.array([\n (x <= np.arange(600)) for x in diastole\n ], dtype=np.uint8)\n[quote=Franc Bračun;103338]
\n\n@Mathurin:
\n\nI had the same problem on Windows. I fixed it by adding one line of code root=root.replace('\\\\','/') and it works for me. See the relevant part of code below. Hope this will help you.
for root, _, files in os.walk(root_path):\n root=root.replace('\\\\','/')\n if len(files) == 0 or not files[0].endswith(".dcm") or root.find("sax") == -1:\n continue\n prefix = files[0].rsplit('-', 1)[0]\n fileset = set(files)\n expected = ["%s-%04d.dcm" % (prefix, i + 1) for i in range(30)]\n if all(x in fileset for x in expected):\n ret.append([root + "/" + x for x in expected])\n # sort for reproduciblity\n return sorted(ret, key = lambda x: x[0])\n[/quote]
\n\nThis seems to be a great solution, maybe you can send a PR back so the future windows users can benefit from it, we can use python's platform check to check platforms
", "rawMarkdown": "[quote=Franc Bračun;103338]\r\n\r\n@Mathurin:\r\n\r\nI had the same problem on Windows. I fixed it by adding one line of code `root=root.replace('\\\\','/')` and it works for me. See the relevant part of code below. Hope this will help you.\r\n\r\n for root, _, files in os.walk(root_path):\r\n root=root.replace('\\\\','/')\r\n if len(files) == 0 or not files[0].endswith(\".dcm\") or root.find(\"sax\") == -1:\r\n continue\r\n prefix = files[0].rsplit('-', 1)[0]\r\n fileset = set(files)\r\n expected = [\"%s-%04d.dcm\" % (prefix, i + 1) for i in range(30)]\r\n if all(x in fileset for x in expected):\r\n ret.append([root + \"/\" + x for x in expected])\r\n # sort for reproduciblity\r\n return sorted(ret, key = lambda x: x[0])\r\n\r\n[/quote]\r\n\r\nThis seems to be a great solution, maybe you can send a PR back so the future windows users can benefit from it, we can use python's platform check to check platforms" }, { "id": 103388, "postDate": "2016-01-02T02:32:00.130Z", "content": "@Mathurin
\n\nIf you put your data in another folder, then you need to modify the code in order to take into account the number of subdirectories in the data path. For example, on Linux, I put the train.csv file in the following folder:\nDATA_PATH = '/home/rick/data/datasciencebowl/Kaggle'
\n\nSo now to access the file, I write:
\n\n\n\n\nwrite_label_csv("./train-label.csv", train_frames, get_label_map(os.path.join(DATA_PATH,"train.csv")))
\n
Since there are an additional four subdirectories in the path name, I change the 3 to 7 in the 3rd line of the "def write_label_csv" funtion:
\n\n index = int(lst[0].split("/")[7])\nI hope this helps!
", "rawMarkdown": "@Mathurin\r\n\r\nIf you put your data in another folder, then you need to modify the code in order to take into account the number of subdirectories in the data path. For example, on Linux, I put the train.csv file in the following folder:\r\nDATA_PATH = '/home/rick/data/datasciencebowl/Kaggle'\r\n\r\nSo now to access the file, I write:\r\n\r\n \r\n> write_label_csv(\"./train-label.csv\", train_frames, get_label_map(os.path.join(DATA_PATH,\"train.csv\")))\r\n\r\n\r\n\r\nSince there are an additional four subdirectories in the path name, I change the 3 to 7 in the 3rd line of the \"def write_label_csv\" funtion:\r\n\r\n index = int(lst[0].split(\"/\")[7])\r\n\r\n\r\n\r\nI hope this helps!" }, { "id": 103347, "postDate": "2016-01-01T11:29:34.913Z", "content": "@Franc\nThank you very much for your answer, everything is ok now.
\n\nHappy new year for you and your family and for all kagglers.
", "rawMarkdown": "@Franc\r\nThank you very much for your answer, everything is ok now.\r\n\r\nHappy new year for you and your family and for all kagglers.\r\n" }, { "id": 103323, "postDate": "2015-12-31T20:03:51.107Z", "content": "Thanks redd for your return. I try this too without success. Could you share your preprocessing.py file ?
", "rawMarkdown": "Thanks redd for your return. I try this too without success. Could you share your preprocessing.py file ?\r\n" }, { "id": 103310, "postDate": "2015-12-31T17:07:57.397Z", "content": "Mathurin - I had that one. Are you in Windows? If so, it has to do with the / in Linux vs. \\\\ in Windows, giving you a bad value in variable index. Might try index = int(lst[0].split("\\\\")[3]) and make some changes in the code to use \\\\ instead of /.
", "rawMarkdown": "Mathurin - I had that one. Are you in Windows? If so, it has to do with the / in Linux vs. \\\\\\ in Windows, giving you a bad value in variable index. Might try index = int(lst[0].split(\"\\\\\\\")[3]) and make some changes in the code to use \\\\\\ instead of /." }, { "id": 103307, "postDate": "2015-12-31T16:53:33.303Z", "content": "Anybody can help me with this code\nWhen I run in Preprocessing.py\nwrite_label_csv("./train-label.csv", train_frames, get_label_map("./data/train.csv"))
\n\n\n\n\n\n\nValueError: invalid literal for int() with base 10
\n
The function in error\ndef write_label_csv(fname, frames, label_map):\n fo = open(fname, "w")\n for lst in frames:\n index = int(lst[0].split("/")[3])\n if label_map != None:\n fo.write(label_map[index])\n else:\n fo.write("%d,0,0\\n" % index)\n fo.close()
\n\nThanks
", "rawMarkdown": "Anybody can help me with this code\r\nWhen I run in Preprocessing.py\r\nwrite_label_csv(\"./train-label.csv\", train_frames, get_label_map(\"./data/train.csv\"))\r\n\r\n>> ValueError: invalid literal for int() with base 10\r\n\r\nThe function in error\r\ndef write_label_csv(fname, frames, label_map):\r\n fo = open(fname, \"w\")\r\n for lst in frames:\r\n index = int(lst[0].split(\"/\")[3])\r\n if label_map != None:\r\n fo.write(label_map[index])\r\n else:\r\n fo.write(\"%d,0,0\\n\" % index)\r\n fo.close()\r\n\r\n\r\nThanks\r\n\r\n\r\n\r\n" }, { "id": 103255, "postDate": "2015-12-30T19:25:28.893Z", "content": "Hi all,
\n\nI have attempted to implement the tutorial, but I am running into the error:
\n\n"objc[58310]: Class CVWindow is implemented in both /usr/local/opt/opencv/lib/libopencv_highgui.2.4.dylib and /Users/ocolegro/anaconda/lib/libopencv_highgui.2.4.8.dylib. One of the two will be used. Which one is undefined.\nobjc[58310]: Class CVView is implemented in both /usr/local/opt/opencv/lib/libopencv_highgui.2.4.dylib and /Users/ocolegro/anaconda/lib/libopencv_highgui.2.4.8.dylib. One of the two will be used. Which one is undefined.\nobjc[58310]: Class CVSlider is implemented in both /usr/local/opt/opencv/lib/libopencv_highgui.2.4.dylib and /Users/ocolegro/anaconda/lib/libopencv_highgui.2.4.8.dylib. One of the two will be used. Which one is undefined.\nobjc[58310]: Class CaptureDelegate is implemented in both /usr/local/opt/opencv/lib/libopencv_highgui.2.4.dylib and /Users/ocolegro/anaconda/lib/libopencv_highgui.2.4.8.dylib. One of the two will be used. Which one is undefined.\n[19:22:21] ./dmlc-core/include/dmlc/logging.h:208: [19:22:21] src/data.cc:43: Unknown data type csv\nTraceback (most recent call last):\n File "train.py", line 85, in \n batch_size=batch_size)\n File "/Users/ocolegro/anaconda/lib/python2.7/site-packages/mxnet-0.5.0-py2.7.egg/mxnet/io.py", line 508, in creator\n ctypes.byref(iter_handle)))\n File "/Users/ocolegro/anaconda/lib/python2.7/site-packages/mxnet-0.5.0-py2.7.egg/mxnet/base.py", line 76, in check_call\n raise MXNetError(py_str(_LIB.MXGetLastError()))\nmxnet.base.MXNetError: [19:22:21] src/data.cc:43: Unknown data type csv\n"
\n\nAm I alone in getting the "Unkown data type csv" error? I did some googling and saw that it is a common problem, but re-pulling the git repo did not fix my issues.
", "rawMarkdown": "Hi all,\r\n\r\nI have attempted to implement the tutorial, but I am running into the error:\r\n\r\n\"objc[58310]: Class CVWindow is implemented in both /usr/local/opt/opencv/lib/libopencv_highgui.2.4.dylib and /Users/ocolegro/anaconda/lib/libopencv_highgui.2.4.8.dylib. One of the two will be used. Which one is undefined.\r\nobjc[58310]: Class CVView is implemented in both /usr/local/opt/opencv/lib/libopencv_highgui.2.4.dylib and /Users/ocolegro/anaconda/lib/libopencv_highgui.2.4.8.dylib. One of the two will be used. Which one is undefined.\r\nobjc[58310]: Class CVSlider is implemented in both /usr/local/opt/opencv/lib/libopencv_highgui.2.4.dylib and /Users/ocolegro/anaconda/lib/libopencv_highgui.2.4.8.dylib. One of the two will be used. Which one is undefined.\r\nobjc[58310]: Class CaptureDelegate is implemented in both /usr/local/opt/opencv/lib/libopencv_highgui.2.4.dylib and /Users/ocolegro/anaconda/lib/libopencv_highgui.2.4.8.dylib. One of the two will be used. Which one is undefined.\r\n[19:22:21] ./dmlc-core/include/dmlc/logging.h:208: [19:22:21] src/data.cc:43: Unknown data type csv\r\nTraceback (most recent call last):\r\n File \"train.py\", line 85, in@waheguru I guess you either run cpu version, e.g. mnist is fast on cpu, or you have an 2014 or older macbook pro which has cuda card. the new 2015 macbook pro has Intel Iris and AMD
", "rawMarkdown": "@waheguru I guess you either run cpu version, e.g. mnist is fast on cpu, or you have an 2014 or older macbook pro which has cuda card. the new 2015 macbook pro has Intel Iris and AMD" }, { "id": 103100, "postDate": "2015-12-28T22:38:48.007Z", "content": "So the thing is that I was able run the mxnet examples (e.g. image classification) on my Mac successfully. Why is that?
\n\n[quote=Özgün Genç;102970]
\n\nNo, only Nvidia GPU's support CUDA, and OpenCl is not supported by mxnet (or any other DL packages). So AWS is the way to go.
\n\n\n\n\nI'm running on a Macbook Pro with an Intel Iris 1536 MB graphics card.\n I installed CUDA. Is there a way I can run on a mac? If not, I was\n planning on using AWS EC2 Ubuntu.
\n
[/quote]
", "rawMarkdown": "So the thing is that I was able run the mxnet examples (e.g. image classification) on my Mac successfully. Why is that?\r\n\r\n[quote=Özgün Genç;102970]\r\n\r\nNo, only Nvidia GPU's support CUDA, and OpenCl is not supported by mxnet (or any other DL packages). So AWS is the way to go.\r\n\r\n> I'm running on a Macbook Pro with an Intel Iris 1536 MB graphics card.\r\n> I installed CUDA. Is there a way I can run on a mac? If not, I was\r\n> planning on using AWS EC2 Ubuntu.\r\n\r\n[/quote]\r\n" }, { "id": 103059, "postDate": "2015-12-28T17:43:51.710Z", "content": "@Tim Hochberg, Thanks for explanation, now I understand. I still want to try my own approach, but it keeps producing bad results. lol
", "rawMarkdown": "@Tim Hochberg, Thanks for explanation, now I understand. I still want to try my own approach, but it keeps producing bad results. lol" }, { "id": 103041, "postDate": "2015-12-28T14:56:58.077Z", "content": "@Bing Xu, that's excellent starter code. Thank you very much.
", "rawMarkdown": "@Bing Xu, that's excellent starter code. Thank you very much. " }, { "id": 103017, "postDate": "2015-12-28T12:09:46.417Z", "content": "mxnet team rocks!!!\nMay I ask that what is the output CRPS of the model on training dataset? Is it roughly equivalent to 0.039 on validation?
", "rawMarkdown": "mxnet team rocks!!!\r\nMay I ask that what is the output CRPS of the model on training dataset? Is it roughly equivalent to 0.039 on validation?" }, { "id": 102970, "postDate": "2015-12-28T05:33:20.010Z", "content": "No, only Nvidia GPU's support CUDA, and OpenCl is not supported by mxnet (or any other DL packages). So AWS is the way to go.
\n\n\n", "rawMarkdown": "No, only Nvidia GPU's support CUDA, and OpenCl is not supported by mxnet (or any other DL packages). So AWS is the way to go.\r\n\r\n> I'm running on a Macbook Pro with an Intel Iris 1536 MB graphics card.\r\n> I installed CUDA. Is there a way I can run on a mac? If not, I was\r\n> planning on using AWS EC2 Ubuntu." }, { "id": 102932, "postDate": "2015-12-27T21:49:14.630Z", "content": "I'm running on a Macbook Pro with an Intel Iris 1536 MB graphics card.\n I installed CUDA. Is there a way I can run on a mac? If not, I was\n planning on using AWS EC2 Ubuntu.
\n
Thanks Özgün, I think that worked! There's one last thing I need to figure out re: CUDA.
\n\nFile "/Users/anaconda/lib/python3.5/site-packages/mxnet-0.5.0-py3.5.egg/mxnet/base.py", line 76, in check_call\n raise MXNetError(py_str(_LIB.MXGetLastError()))\nmxnet.base.MXNetError: [16:42:46] src/storage/storage.cc:44: Please compile with CUDA enabled
\n\nI'm running on a Macbook Pro with an Intel Iris 1536 MB graphics card. I installed CUDA. Is there a way I can run on a mac? If not, I was planning on using AWS EC2 Ubuntu.
\n\n[quote=Özgün Genç;102913]
\n\nYou need to reinstall the python package as well.
\n\ncd python; sudo python setup.py install\n[/quote]
", "rawMarkdown": "Thanks Özgün, I think that worked! There's one last thing I need to figure out re: CUDA. \r\n\r\n File \"/Users/anaconda/lib/python3.5/site-packages/mxnet-0.5.0-py3.5.egg/mxnet/base.py\", line 76, in check_call\r\n raise MXNetError(py_str(_LIB.MXGetLastError()))\r\nmxnet.base.MXNetError: [16:42:46] src/storage/storage.cc:44: Please compile with CUDA enabled\r\n\r\nI'm running on a Macbook Pro with an Intel Iris 1536 MB graphics card. I installed CUDA. Is there a way I can run on a mac? If not, I was planning on using AWS EC2 Ubuntu. \r\n\r\n[quote=Özgün Genç;102913]\r\n\r\nYou need to reinstall the python package as well. \r\n\r\n cd python; sudo python setup.py install\r\n\r\n[/quote]\r\n" }, { "id": 102917, "postDate": "2015-12-27T18:43:51.260Z", "content": "It still errors out even it has been changed to fi.next(), see below:
\n\n$ python2.7 Train.py\nTraceback (most recent call last):\n File "Train.py", line 88, in <module>\n batch_size=1)\n File "/usr/local/lib/python2.7/dist-packages/mxnet-0.5.0-py2.7.egg/mxnet/io.py", line 513, in creator\n return MXDataIter(iter_handle, **kwargs)\n File "/usr/local/lib/python2.7/dist-packages/mxnet-0.5.0-py2.7.egg/mxnet/io.py", line 376, in __init__\n self.first_batch = self.next()\n File "/usr/local/lib/python2.7/dist-packages/mxnet-0.5.0-py2.7.egg/mxnet/io.py", line 419, in next\n raise StopIteration\nStopIteration\n[quote=dotcoming;102691]
\n\nThanks for this project. \nI have already used it successfully after building mxnet.
\n\nAnd I found a bug maybe.\nWhen you use python2, line 199 in Train.py should be: fo.writerow(fi.next()).
\n\nThanks again.
\n\n[/quote]
", "rawMarkdown": "It still errors out even it has been changed to fi.next(), see below:\r\n\r\n $ python2.7 Train.py\r\n Traceback (most recent call last):\r\n File \"Train.py\", line 88, inYou need to reinstall the python package as well.
\n\ncd python; sudo python setup.py install\nThanks for the response. I tried to do that, but now receive similar errors:
\n\nAttributeError: module 'mxnet' has no attribute 'symbol'
\n\n[quote=Özgün Genç;102873]
\n\n@Waheguru This means your mxnet module doesn't have the mx.sym attribute. What happens if you replace all mx.sym with mx.symbol?
\n\n[/quote]
", "rawMarkdown": "Thanks for the response. I tried to do that, but now receive similar errors:\r\n\r\nAttributeError: module 'mxnet' has no attribute 'symbol'\r\n\r\n\r\n[quote=Özgün Genç;102873]\r\n\r\n@Waheguru This means your mxnet module doesn't have the mx.sym attribute. What happens if you replace all mx.sym with mx.symbol?\r\n\r\n[/quote]\r\n" }, { "id": 102908, "postDate": "2015-12-27T17:45:09.167Z", "content": "Thanks for the repsonse. I just deleted mxnet and ran
\n\ngit clone --recursive https://github.com/dmlc/mxnet\ncd mxnet; make -j4
\n\nStill getting the same error. Anything else I should try?
\n\n[quote=Alessandro Mariani;102877]
\n\n[quote=Waheguru;102786]
\n\nI'm able to run Preprocessing successfully, but then when trying to Train the data, I get
\n\nTraceback (most recent call last):\n File "Train.py", line 77, in \n network = get_lenet()\n File "Train.py", line 16, in get_lenet\n source = mx.sym.Variable("data")\nAttributeError: 'module' object has no attribute 'sym'
\n\nAnyone have any pointers on this? Thanks.
\n\n[/quote]
\n\nyour mxnet is outdated, so you need to rebuild and reinstall mxnet!
\n\n[/quote]
", "rawMarkdown": "Thanks for the repsonse. I just deleted mxnet and ran\r\n\r\ngit clone --recursive https://github.com/dmlc/mxnet\r\ncd mxnet; make -j4\r\n\r\nStill getting the same error. Anything else I should try?\r\n\r\n\r\n[quote=Alessandro Mariani;102877]\r\n\r\n[quote=Waheguru;102786]\r\n\r\nI'm able to run Preprocessing successfully, but then when trying to Train the data, I get \r\n\r\nTraceback (most recent call last):\r\n File \"Train.py\", line 77, in[quote=Waheguru;102786]
\n\nI'm able to run Preprocessing successfully, but then when trying to Train the data, I get
\n\nTraceback (most recent call last):\n File "Train.py", line 77, in \n network = get_lenet()\n File "Train.py", line 16, in get_lenet\n source = mx.sym.Variable("data")\nAttributeError: 'module' object has no attribute 'sym'
\n\nAnyone have any pointers on this? Thanks.
\n\n[/quote]
\n\nyour mxnet is outdated, so you need to rebuild and reinstall mxnet!
", "rawMarkdown": "[quote=Waheguru;102786]\r\n\r\nI'm able to run Preprocessing successfully, but then when trying to Train the data, I get \r\n\r\nTraceback (most recent call last):\r\n File \"Train.py\", line 77, in@Waheguru This means your mxnet module doesn't have the mx.sym attribute. What happens if you replace all mx.sym with mx.symbol?
", "rawMarkdown": "@Waheguru This means your mxnet module doesn't have the mx.sym attribute. What happens if you replace all mx.sym with mx.symbol?" }, { "id": 102691, "postDate": "2015-12-24T13:39:05.923Z", "content": "Thanks for this project. \nI have already used it successfully after building mxnet.
\n\nAnd I found a bug maybe.\nWhen you use python2, line 199 in Train.py should be: fo.writerow(fi.next()).
\n\nThanks again.
", "rawMarkdown": "Thanks for this project. \r\nI have already used it successfully after building mxnet.\r\n \r\nAnd I found a bug maybe.\r\n**When you use python2, line 199 in Train.py should be: fo.writerow(fi.next()).**\r\n\r\nThanks again." }, { "id": 102660, "postDate": "2015-12-24T08:04:34.337Z", "content": "I had this one with the processing part: switching from absolute to relative path helped. The splitting part of the corresponding function does not expect strings after a certain point in the split path.
", "rawMarkdown": "I had this one with the processing part: switching from absolute to relative path helped. The splitting part of the corresponding function does not expect strings after a certain point in the split path." }, { "id": 102655, "postDate": "2015-12-24T07:28:26.300Z", "content": "ValueError: invalid literal for int() with base 10: 'cb552bf1-c649-4cca-8aca-3c24afca817b'
\n\nI got an error like above when compile index = int(lst[0].split("/")[3])
", "rawMarkdown": "ValueError: invalid literal for int() with base 10: 'cb552bf1-c649-4cca-8aca-3c24afca817b'\r\n\r\nI got an error like above when compile index = int(lst[0].split(\"/\")[3])" }, { "id": 102601, "postDate": "2015-12-23T20:23:34.850Z", "content": "I remember Eric has marco to disable RTC for Cuda 6. Please raise issue directly in MXNet for installing problem instead of here.
\n\nThanks.
\n\n[quote=Mujtaba Hasan;102534]
\n\n[quote=Bing Xu;102513]
\n\nshould be ok but not tested\n[quote=Mujtaba Hasan;102508]
\n\ndoes it work with cuda 6.5
\n\n[/quote]
\n\n[/quote]
\n\nIt can't find nvrtc.h\n"""\n/home/mujtaba/Documents/mxnet/include/mxnet/mxrtc.h:12:19: fatal error: nvrtc.h: No such file or directory\n #include \n ^\ncompilation terminated.\n"""
\n\n[/quote]
", "rawMarkdown": "I remember Eric has marco to disable RTC for Cuda 6. Please raise issue directly in MXNet for installing problem instead of here. \r\n\r\nThanks.\r\n\r\n[quote=Mujtaba Hasan;102534]\r\n\r\n[quote=Bing Xu;102513]\r\n\r\nshould be ok but not tested\r\n[quote=Mujtaba Hasan;102508]\r\n\r\ndoes it work with cuda 6.5\r\n\r\n\r\n[/quote]\r\n\r\n\r\n[/quote]\r\n\r\nIt can't find nvrtc.h\r\n\"\"\"\r\n/home/mujtaba/Documents/mxnet/include/mxnet/mxrtc.h:12:19: fatal error: nvrtc.h: No such file or directory\r\n #include@Mujtaba Hasan have you included your CUDA in the path or put it in the config.mk ADD_FLAGS?
", "rawMarkdown": "@Mujtaba Hasan have you included your CUDA in the path or put it in the config.mk ADD_FLAGS?" }, { "id": 102534, "postDate": "2015-12-23T13:00:08.637Z", "content": "[quote=Bing Xu;102513]
\n\nshould be ok but not tested\n[quote=Mujtaba Hasan;102508]
\n\ndoes it work with cuda 6.5
\n\n[/quote]
\n\n[/quote]
\n\nIt can't find nvrtc.h\n"""\n/home/mujtaba/Documents/mxnet/include/mxnet/mxrtc.h:12:19: fatal error: nvrtc.h: No such file or directory\n #include \n ^\ncompilation terminated.\n"""
", "rawMarkdown": "[quote=Bing Xu;102513]\r\n\r\nshould be ok but not tested\r\n[quote=Mujtaba Hasan;102508]\r\n\r\ndoes it work with cuda 6.5\r\n\r\n\r\n[/quote]\r\n\r\n\r\n[/quote]\r\n\r\nIt can't find nvrtc.h\r\n\"\"\"\r\n/home/mujtaba/Documents/mxnet/include/mxnet/mxrtc.h:12:19: fatal error: nvrtc.h: No such file or directory\r\n #includeshould be ok but not tested\n[quote=Mujtaba Hasan;102508]
\n\ndoes it work with cuda 6.5
\n\n[/quote]
", "rawMarkdown": "should be ok but not tested\r\n[quote=Mujtaba Hasan;102508]\r\n\r\ndoes it work with cuda 6.5\r\n\r\n\r\n[/quote]\r\n" }, { "id": 102508, "postDate": "2015-12-23T06:21:04.070Z", "content": "does it work with cuda 6.5
", "rawMarkdown": "does it work with cuda 6.5\r\n" }, { "id": 102437, "postDate": "2015-12-22T17:05:22.937Z", "content": "yes Python 2.7.11
", "rawMarkdown": "yes Python 2.7.11" }, { "id": 104478, "postDate": "2016-01-13T04:27:23.470Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 103814, "author_name": "Tim Hochberg", "author_url": "", "post_date": "2016-01-06T18:32:27.367000", "content": "@EIGSI, I don't want to give away too many secrets, but here are some general suggestions:
\n\nNumbers 1 and 2 are pretty standard for neural nets and I nearly always use them. Number 3 is always worth trying, it's fairly quick to try and can sometimes have a large payoff.
\n\nI wouldn't worry about learning rate or momentum tuning till you fix your overfitting issues. All you would likely be able to do is to increase the amount of overfit. Averaging the results of cross validation is a perfectly reasonable idea, but likely very, very slow. It's not something I'd try till I'd completely run out of other ideas.
\n\nGood luck!
", "votes": 9, "replies": [] }, { "id": 102448, "author_name": "phunter", "author_url": "", "post_date": "2015-12-22T18:11:07.710000", "content": "thumbs up for MXnet!
\n\nFor people who want to install GPU version of MXnet, please refer to this blog https://no2147483647.wordpress.com/2015/12/07/deep-learning-for-hackers-with-mxnet-1/
\n\nPlus a fun of neural art with MXnet: https://no2147483647.wordpress.com/2015/12/21/deep-learning-for-hackers-with-mxnet-2/
", "votes": 8, "replies": [] }, { "id": 103815, "author_name": "phunter", "author_url": "", "post_date": "2016-01-06T18:46:25.320000", "content": "@WD you are welcome. Yes, AWS GPU instance needs some kernel image update for installing CUDA if ones wants to install from Ubuntu scratch (https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN) ) I will make a MXnet version for it. Or, one can use docker anyway http://mxnt.ml/en/latest/build.html#docker-images
\n\nAnd, 'star-ed' means 'click the star button on the right top and add a star to MXnet github' :-) I am wondering why the slow tensorflow has 10k+ stars on github while all core code pull requests being rejected (you call it open source?), but the faster, easy-to-use, real open source MXnet only has 2000+ stars now. Please click the star button https://github.com/dmlc/mxnet
\n\nThe reasons why MXnet is good for competitions:
\n\n@EIGSI some of my tricks/techniques:
\n\nThank you so much for sharing (and advancing mxnet)!\nI had a small bug which is in the line
\n\n img = preproc(f.pixel_array / np.max(f.pixel_array))\nThe problem is that for me, dicom, returns int pixels so dividing by the max returns an all zero image (except for the max) so a possible fix is:
img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\nThe easiest should use SFrame.\nI will write a tutorial on how to use SFrame with MXNet. Hopefully in this week.\nhttps://github.com/dmlc/mxnet/tree/master/plugin/sframe
\n\nhttps://github.com/dato-code/SFrame
\n\n[quote=WD;105654]
\n\n@phunter. thanks for this. i will look into both the CSV route and the Io.record route. in the CSV approach - i guess the easiest way would be to work with the numpy "img" object - no? i will play around with this and make code public if i find something useful to others
\n\n f = dicom.read_file(path)\n img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\n dst_path = path.rsplit(".", 1)[0] + ".64x64.jpg" #create jpg out of all files\n[/quote]
", "votes": 4, "replies": [] }, { "id": 106773, "author_name": "Franc Bračun", "author_url": "", "post_date": "2016-02-03T20:58:14.067000", "content": "[quote=WD;106742]
\n\n@Franc. thanks so much. should one add as well a convolutional relu layer between the flatten and the fc1? as per below? or is that not correct / needed?
\n\n flatten = mx.symbol.Flatten(net)\n flatten = mx.symbol.Dropout(flatten)\n flatten = mx.sym.Activation(flatten, act_type="relu") ###to be checked\n fc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=1024)\n fc1 = mx.sym.Activation(fc1, act_type="relu")\n fc1 = mx.symbol.Dropout(fc1)\n fc2 = mx.symbol.FullyConnected(data=fc1, num_hidden=1)\n[/quote]\nNo, usually you do not put convolutinal layer between flatten and FC layer. Notice, that in the third line in the code above you have only activation layer. I would simply remove the third line of code, i.e. flatten = mx.sym.Activation(flatten, act_type="relu")
I just want to stick this out here for those still stuck trying to process on Windows with Python 3.5. The issue with the processing file failing at on the last line is because when using csv.writer on a Windows machine it will add a extra line to each row of data. To fix this error open the file that you want to write to with open(file, "w", newline = ''). Many thanks to Scott Smith for telling me this almost of month ago. Sad thing is I just realized it today after I fixed the code.
", "votes": 1, "replies": [] }, { "id": 106343, "author_name": "WD", "author_url": "", "post_date": "2016-01-30T13:10:15.347000", "content": "@Linear regression. Tooled the Mxnet tutoral over from logistic to linear regression. happy to share code if of interest to others
", "votes": 1, "replies": [] }, { "id": 106251, "author_name": "WD", "author_url": "", "post_date": "2016-01-29T12:29:29.980000", "content": "@Tim. At a more elementary level, i was thinking, that if the network outputs (on e.g. a hypothetical 8 output notes, labelled from 1 to 8): 0.1, 0.1, 0.1, 0.1, 0.99, 0.99, 0.99, 0.99.
\n\nThen the Mxnet would create a CDF that would be 0.1, 0.1, 0.1, 0.1, 0.99, 0.99, 0.99, 0.99. This is almost a step function at node 5.
\n\nHowever, as the network is equally confident that the output might be 5,6,7 and 8, a naieve best guess on best estimator would potentially be the average between 5 to 8, e.g. 6.5
\n\n(normalizing the inputs and subsequently calculating a CDF of e.g. (0.025,0.05,0.075,0.1,0.325,0.55,0.775,1) also does intuitively not make sense)
\n\nHence - I was wondering whether there is an opportunity / need to adjust the CDF that the Mxnet tutoral calculates? I realize that this is a trivial example - but still would love to get your thoughts.
", "votes": 1, "replies": [] }, { "id": 106079, "author_name": "DavidGbodiOdaibo", "author_url": "", "post_date": "2016-01-28T10:54:50.850000", "content": "@Bartek that's the beauty of neural networks they capture the "je ne sais quoi" (I DONT KNOW WHAT) in the data :). What makes you able to distinguish faces of 2 different people, you can't really explain why? There are some obvious things you see but there are latent features baked into your brain that make you able to distinguish faces. That's what neural networks capture. The technical explanation is that they capture hierarchical representations of features in the data. In my opinion the first tutorial is trying to explain why 2 faces are different (segmenting e.t.c) there are just too many variables, scale, noise, orientation e.t.c for it to do a good job. Early in the competition people who tried that approach reported scores of 0.1xxxx which is very bad compared to the current scores on the leaderboard.
", "votes": 1, "replies": [] }, { "id": 106024, "author_name": "Tim Hochberg", "author_url": "", "post_date": "2016-01-27T23:16:26.087000", "content": "@WD, that's an interesting point. If my quick scribblings are correct, then computing the MSE on the CDF (versus a step function) is equivalent to computing a weighted MSE on the PDF. The weights would be linearly decreasing[1]. So, for instance the weights could be 600, 599, ...., 2, 1. Thus, if one is outputting the CDF directly one is putting more emphasis on the lower values than upper values. This makes intuitive sense, since an incorrect early value in the PDF will result in all subsequent values of the CDF being off.
\n\nI would suspect that computing the CDF directly will result in better scores than computing the PDF (using softmax) and then converting that to CDF since CDF is the eventual target. I may be wrong about that though, so I encourage you to try it.
\n\n[EDIT] OK, I looked at it a bit more and that math is wrong, but I think the conclusion is probably more or less right, earlier points end up being weighted more than later points.
\n\n[1] Whether the weights are linear or quadratic depends on whether one is applying them before or after squaring. Here I assume that they are applied before squaring.
", "votes": 1, "replies": [] }, { "id": 106009, "author_name": "WD", "author_url": "", "post_date": "2016-01-27T22:07:55.113000", "content": "@Tim. As a follow-up thought - it seems that the indepedent sigmoid approach and the resulting CDF approximation creates a CDF that is a little "biased to the left". A actual softmax "adjustment" and proper CDF calculation from these values might lead to an improvement in scores. will try this out later.
", "votes": 1, "replies": [] }, { "id": 105964, "author_name": "Tim Hochberg", "author_url": "", "post_date": "2016-01-27T18:41:00.023000", "content": "@WD, even though this layer is labelled "softmax", I believe it's just a sigmoid output. The net is supposed to figure out that the output should look like a CDF during training, so the output should be approximately ascending, but it can have glitches. That is why there is submission_helper, it fixes up any non-monotonicity.
Hope that helps.
", "votes": 1, "replies": [] }, { "id": 105654, "author_name": "WD", "author_url": "", "post_date": "2016-01-25T16:46:54.887000", "content": "@phunter. thanks for this. i will look into both the CSV route and the Io.record route. in the CSV approach - i guess the easiest way would be to work with the numpy "img" object - no? i will play around with this and make code public if i find something useful to others
\n\n f = dicom.read_file(path)\n img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\n dst_path = path.rsplit(".", 1)[0] + ".64x64.jpg" #create jpg out of all files\nThere is rotation and a bunch of augmentation method, but require ImageRecord IO. For this example, we only want to show how to use MXNet so we choose CSV.
\n\nhttps://mxnet.readthedocs.org/en/latest/python/io.html#mxnet.io.ImageRecordIter
\n\n[quote=phunter;105650]
\n\nthe mxnet tutorial code has not (yet) support image rotation. one can modify preprocessing.py and generate CSV with rotation by scikit-image
\n\n[/quote]
", "votes": 1, "replies": [] }, { "id": 105164, "author_name": "WD", "author_url": "", "post_date": "2016-01-20T15:34:57.017000", "content": "G2x2 spot runs at around 0.1$ per hour. Regular instances (not spot) are more expensive - 0.65$. it depends on your launch zone. The price graphs can be found on AWS - when go to spot instances.
", "votes": 1, "replies": [] }, { "id": 105066, "author_name": "WD", "author_url": "", "post_date": "2016-01-19T18:09:39.490000", "content": "I am finishing up code to extract and plot training and validation curves, and I will share this code today or tomorrow. I had a quick question. If one uses a terminal mode to access one's AWS instance (e.g. using Putty) and either runs scripts from the bash command line or from an ipython shell, then is it possible to plot graphs visually? or does one either a) have to install an ipython notebook, or b) save-to-image, transfer the files back to one's local computer, and plot locally?
", "votes": 1, "replies": [] }, { "id": 105013, "author_name": "WD", "author_url": "", "post_date": "2016-01-18T20:33:18.983000", "content": "i am trying to visualize the network. Some others might be interested in this as well. I thought the code below should do the trick. However, even doing all the sudo apt-get installs (ubuntu) i still get a graphviz library not found error. let me know if anyone has experience with this or has encountered the same issue or knows how to resolve this!
\n\n\n", "votes": 1, "replies": [] }, { "id": 104979, "author_name": "WD", "author_url": "", "post_date": "2016-01-18T15:06:23.807000", "content": "import graphviz import find_mxnet import mxnet as mx import importlib
\n \ndef get_lenet():\n #the code from the train.py file
\n \nnetwork = get_lenet()
\n \nmx.viz.plot_network(network)
\n
@EIGSI - you can also log the validation curves by using e.g. stytole_model.fit(X=data_train, eval_metric = mx.metric.np(CRPS),batch_end_callback = mx.callback.log_train_metric(32))
", "votes": 1, "replies": [] }, { "id": 104955, "author_name": "Franc Bračun", "author_url": "", "post_date": "2016-01-18T08:47:26.240000", "content": "@Jon Carlies: since precompiled mxnet libraries are only cpu enabled, replace all : devs = [mx.gpu(0)] with devs = [mx.cpu()].
[quote=Scott Smith;104308]
\n\n[quote=Matthew Tubs;104174]
\n\nUpdate: I got the code running smoother now. Turned out that while messing with R I made images of all the dicom files and that was interrupting the code.
\n\nHowever, I have ran into a different error at the end of the run. Can anyone help me figure why this index went out of range?
\n\nif split_to_train[cnt]:
\n\nIndexError: list index out of range
\n\n[/quote]
\n\n@Matthew: I think I ran into the same issues preprocessing with Python 3.5 on Windows.
\n\n[/quote]
\n\nI'm on the last line of code and reviewing #2 in this post to try and solve my index issue. For me, he train_frames has a length of 5,293 yet train-64x64-data.csv has double that which I believe is why there is an index error. Am I understanding this correctly...how many rows should I have in train-64x64-data.csv?
", "votes": 1, "replies": [] }, { "id": 104308, "author_name": "Scott Smith", "author_url": "", "post_date": "2016-01-11T16:16:21.163000", "content": "[quote=Matthew Tubs;104174]
\n\nUpdate: I got the code running smoother now. Turned out that while messing with R I made images of all the dicom files and that was interrupting the code.
\n\nHowever, I have ran into a different error at the end of the run. Can anyone help me figure why this index went out of range?
\n\nif split_to_train[cnt]:
\n\nIndexError: list index out of range
\n\n[/quote]
\n\n@Matthew: I think I ran into the same issues preprocessing with Python 3.5 on Windows.
\n\n[quote=Matthew Tubs;104166]
\n\n[quote=earino;104164]
\n\n[quote=Matthew Tubs;104163]\nFranc
\n\nI've been trying to get the Preprocessing.py file you attached running and I've ran into a issue. Would you help me out in understanding what is going on? I am running Python 3.5 on windows, and I am receiving the following error.
\n\ntran_index = np.loadtxt("./tran_label.csv", delimiter=",")[:,0].astype("int")
\n\nIndexError: too many indices for array
\n\nThanks much\n [/quote]
\n\nis that tran_label supposed to be train_label?
\n\n[/quote]\nI did a replace all to see if the problem was something attached to the name of the file.
\n\n[/quote]
\n\nUpdate: I got the code running smoother now. Turned out that while messing with R I made images of all the dicom files and that was interrupting the code.
\n\nHowever, I have ran into a different error at the end of the run. Can anyone help me figure why this index went out of range?
\n\nif split_to_train[cnt]:
\n\nIndexError: list index out of range
", "votes": 1, "replies": [] }, { "id": 104116, "author_name": "earino", "author_url": "", "post_date": "2016-01-09T05:38:46.777000", "content": "Hi! What is the license of your tutorial? I don't see one posted in the github?
", "votes": 1, "replies": [] }, { "id": 104026, "author_name": "WD", "author_url": "", "post_date": "2016-01-08T16:27:52.357000", "content": "i received an odd error running processing.py. what might be the root-cause?
\n\nW
\n\n800 slices processed\nTraceback (most recent call last):\n File "Preprocessing.py", line 134, in \n valid_lst = write_data_csv("./validate-64x64-data.csv", validate_frames, lambda x: crop_resize(x, 64))\n File "Preprocessing.py", line 66, in write_data_csv\n img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\n File "/usr/lib/python2.7/dist-packages/dicom/dataset.py", line 405, in _get_pixel_array\n return self._getPixelArray()\n File "/usr/lib/python2.7/dist-packages/dicom/dataset.py", line 400, in _getPixelArray\n self._PixelArray = self._PixelDataNumpy()\n File "/usr/lib/python2.7/dist-packages/dicom/dataset.py", line 382, in _PixelDataNumpy\n arr = arr.reshape(self.Rows, self.Columns)\nValueError: total size of new array must be unchanged\nubuntu@ip-172-31-7-91:~$
", "votes": 1, "replies": [] }, { "id": 103808, "author_name": "", "author_url": "", "post_date": "2016-01-06T18:07:47.057000", "content": "The figure shows my best submission so far using this code (0.038332). I am using the local train/test datasets generated by Preprocessing.py (1/10th for local test, rest is for training). Changed the epoch to 100 for both networks. I also had to reduce batch_size to 8 due to my low-memory GPU.
\n\nFor the sake of learning, what else can we do to improve this code's performance? Playing with the learning rate, momentum? Doing 10-fold cross-validation and averaging predictions of those 10 models?
", "votes": 1, "replies": [] }, { "id": 103672, "author_name": "Keiku", "author_url": "", "post_date": "2016-01-05T12:50:38.567000", "content": "Train.py code worked so well, but mx.model.FeedForward.create in Train.R code does not work. What is the cause?
\n\n> # Training the stytole net\n> mx.set.seed(0)\n> stytole_model <- mx.model.FeedForward.create(\n+ X = data_train,\n+ ctx = mx.gpu(0),\n+ symbol = network,\n+ num.round = 65,\n+ learning.rate = 0.001,\n+ wd = 0.00001,\n+ momentum = 0.9,\n+ eval.metric = mx.metric.CRPS\n+ )\nStart training with 1 devices\n[1] Train-CRPS=0.248537092806794\n[2] Train-CRPS=0.249575979181468\n[3] Train-CRPS=0.249281826982739\n[4] Train-CRPS=0.248988226596111\n[5] Train-CRPS=0.248694746602638\n[6] Train-CRPS=0.248402110901456\n[7] Train-CRPS=0.248110486441219\n[8] Train-CRPS=0.247818248773282\n[9] Train-CRPS=0.247526859680184\n[10] Train-CRPS=0.247235201653695\n(Omitted)\n[55] Train-CRPS=0.234554654820709\n[56] Train-CRPS=0.234282698395829\n[57] Train-CRPS=0.234008295194647\n[58] Train-CRPS=0.233738730834488\n[59] Train-CRPS=0.233467098573434\n[60] Train-CRPS=0.233195344441995\n[61] Train-CRPS=0.232924924355617\n[62] Train-CRPS=0.232653180647331\n[63] Train-CRPS=0.2323852445735\n[64] Train-CRPS=0.23211576977677\n[65] Train-CRPS=0.231845338545091\n@Bing Xu Thanks for this code and introducing to mxnet!
", "votes": 1, "replies": [] }, { "id": 103576, "author_name": "", "author_url": "", "post_date": "2016-01-04T13:48:07.137000", "content": "", "votes": 1, "replies": [] }, { "id": 103338, "author_name": "", "author_url": "", "post_date": "2016-01-01T07:35:33.107000", "content": "", "votes": 1, "replies": [] }, { "id": 103042, "author_name": "", "author_url": "", "post_date": "2015-12-28T14:58:31.917000", "content": "", "votes": 1, "replies": [] }, { "id": 103030, "author_name": "", "author_url": "", "post_date": "2015-12-28T13:56:32.767000", "content": "", "votes": 1, "replies": [] }, { "id": 102786, "author_name": "", "author_url": "", "post_date": "2015-12-26T01:16:48.217000", "content": "", "votes": 1, "replies": [] }, { "id": 102723, "author_name": "", "author_url": "", "post_date": "2015-12-25T00:31:21.047000", "content": "", "votes": 1, "replies": [] }, { "id": 102716, "author_name": "", "author_url": "", "post_date": "2015-12-24T23:01:01.570000", "content": "", "votes": 1, "replies": [] }, { "id": 102495, "author_name": "", "author_url": "", "post_date": "2015-12-23T03:44:40.667000", "content": "", "votes": 1, "replies": [] }, { "id": 102473, "author_name": "", "author_url": "", "post_date": "2015-12-22T22:47:02.917000", "content": "", "votes": 1, "replies": [] }, { "id": 102435, "author_name": "", "author_url": "", "post_date": "2015-12-22T17:00:44.230000", "content": "", "votes": 1, "replies": [] }, { "id": 109395, "author_name": "", "author_url": "", "post_date": "2016-02-25T18:23:03.380000", "content": "", "votes": 2, "replies": [] }, { "id": 109382, "author_name": "", "author_url": "", "post_date": "2016-02-25T16:08:11.683000", "content": "", "votes": 2, "replies": [] }, { "id": 105977, "author_name": "", "author_url": "", "post_date": "2016-01-27T19:25:08.497000", "content": "", "votes": 2, "replies": [] }, { "id": 105619, "author_name": "", "author_url": "", "post_date": "2016-01-25T09:38:00.800000", "content": "", "votes": 2, "replies": [] }, { "id": 105458, "author_name": "", "author_url": "", "post_date": "2016-01-23T07:24:12.130000", "content": "", "votes": 2, "replies": [] }, { "id": 104891, "author_name": "", "author_url": "", "post_date": "2016-01-17T19:16:46.920000", "content": "", "votes": 2, "replies": [] }, { "id": 104850, "author_name": "", "author_url": "", "post_date": "2016-01-17T10:54:23.020000", "content": "", "votes": 2, "replies": [] }, { "id": 104313, "author_name": "", "author_url": "", "post_date": "2016-01-11T17:11:19.327000", "content": "", "votes": 2, "replies": [] }, { "id": 104299, "author_name": "", "author_url": "", "post_date": "2016-01-11T15:41:09.757000", "content": "", "votes": 2, "replies": [] }, { "id": 104072, "author_name": "", "author_url": "", "post_date": "2016-01-08T23:34:14.080000", "content": "", "votes": 2, "replies": [] }, { "id": 103854, "author_name": "", "author_url": "", "post_date": "2016-01-07T03:25:56.470000", "content": "", "votes": 2, "replies": [] }, { "id": 103816, "author_name": "", "author_url": "", "post_date": "2016-01-06T18:47:36.037000", "content": "", "votes": 2, "replies": [] }, { "id": 103813, "author_name": "", "author_url": "", "post_date": "2016-01-06T18:28:48.510000", "content": "", "votes": 2, "replies": [] }, { "id": 103532, "author_name": "", "author_url": "", "post_date": "2016-01-04T02:15:56.277000", "content": "", "votes": 2, "replies": [] }, { "id": 102439, "author_name": "", "author_url": "", "post_date": "2015-12-22T17:07:29.550000", "content": "", "votes": 2, "replies": [] }, { "id": 106262, "author_name": "", "author_url": "", "post_date": "2016-01-29T14:56:53.353000", "content": "", "votes": -1, "replies": [] }, { "id": 102712, "author_name": "", "author_url": "", "post_date": "2015-12-24T21:47:30.247000", "content": "", "votes": -1, "replies": [] }, { "id": 349300, "author_name": "", "author_url": "", "post_date": "2018-06-28T01:57:15.393000", "content": "", "votes": 0, "replies": [] }, { "id": 124769, "author_name": "", "author_url": "", "post_date": "2016-06-22T06:28:46.110000", "content": "", "votes": 0, "replies": [] }, { "id": 110273, "author_name": "", "author_url": "", "post_date": "2016-03-04T08:23:54.913000", "content": "", "votes": 0, "replies": [] }, { "id": 110254, "author_name": "", "author_url": "", "post_date": "2016-03-04T02:18:38.237000", "content": "", "votes": 0, "replies": [] }, { "id": 110253, "author_name": "", "author_url": "", "post_date": "2016-03-04T02:07:06.043000", "content": "", "votes": 0, "replies": [] }, { "id": 110249, "author_name": "", "author_url": "", "post_date": "2016-03-04T01:49:30.213000", "content": "", "votes": 0, "replies": [] }, { "id": 110243, "author_name": "", "author_url": "", "post_date": "2016-03-04T01:05:13.450000", "content": "", "votes": 0, "replies": [] }, { "id": 110237, "author_name": "", "author_url": "", "post_date": "2016-03-04T00:24:29.483000", "content": "", "votes": 0, "replies": [] }, { "id": 110111, "author_name": "", "author_url": "", "post_date": "2016-03-03T01:50:55.807000", "content": "", "votes": 0, "replies": [] }, { "id": 110026, "author_name": "", "author_url": "", "post_date": "2016-03-02T07:07:34.693000", "content": "", "votes": 0, "replies": [] }, { "id": 110022, "author_name": "", "author_url": "", "post_date": "2016-03-02T05:03:24.373000", "content": "", "votes": 0, "replies": [] }, { "id": 109385, "author_name": "", "author_url": "", "post_date": "2016-02-25T16:14:06.077000", "content": "", "votes": 0, "replies": [] }, { "id": 109365, "author_name": "", "author_url": "", "post_date": "2016-02-25T14:14:50.463000", "content": "", "votes": 0, "replies": [] }, { "id": 109332, "author_name": "", "author_url": "", "post_date": "2016-02-25T05:15:59.357000", "content": "", "votes": 0, "replies": [] }, { "id": 109331, "author_name": "", "author_url": "", "post_date": "2016-02-25T05:14:53.783000", "content": "", "votes": 0, "replies": [] }, { "id": 109323, "author_name": "", "author_url": "", "post_date": "2016-02-25T03:28:23.897000", "content": "", "votes": 0, "replies": [] }, { "id": 109313, "author_name": "", "author_url": "", "post_date": "2016-02-25T01:49:47.860000", "content": "", "votes": 0, "replies": [] }, { "id": 109312, "author_name": "", "author_url": "", "post_date": "2016-02-25T01:48:03.200000", "content": "", "votes": 0, "replies": [] }, { "id": 109310, "author_name": "", "author_url": "", "post_date": "2016-02-25T01:30:18.867000", "content": "", "votes": 0, "replies": [] }, { "id": 109249, "author_name": "", "author_url": "", "post_date": "2016-02-24T15:35:33.773000", "content": "", "votes": 0, "replies": [] }, { "id": 109084, "author_name": "", "author_url": "", "post_date": "2016-02-23T05:25:53.830000", "content": "", "votes": 0, "replies": [] }, { "id": 108892, "author_name": "", "author_url": "", "post_date": "2016-02-21T05:55:28.593000", "content": "", "votes": 0, "replies": [] }, { "id": 108872, "author_name": "", "author_url": "", "post_date": "2016-02-21T00:45:56.003000", "content": "", "votes": 0, "replies": [] }, { "id": 106778, "author_name": "", "author_url": "", "post_date": "2016-02-03T21:45:29.960000", "content": "", "votes": 0, "replies": [] }, { "id": 106753, "author_name": "", "author_url": "", "post_date": "2016-02-03T17:32:36.343000", "content": "", "votes": 0, "replies": [] }, { "id": 106742, "author_name": "", "author_url": "", "post_date": "2016-02-03T16:11:58.637000", "content": "", "votes": 0, "replies": [] }, { "id": 106716, "author_name": "", "author_url": "", "post_date": "2016-02-03T13:14:41.283000", "content": "", "votes": 0, "replies": [] }, { "id": 106519, "author_name": "", "author_url": "", "post_date": "2016-02-01T18:01:53.370000", "content": "", "votes": 0, "replies": [] }, { "id": 106432, "author_name": "", "author_url": "", "post_date": "2016-01-31T17:14:48.053000", "content": "", "votes": 0, "replies": [] }, { "id": 106192, "author_name": "", "author_url": "", "post_date": "2016-01-29T01:02:28.703000", "content": "", "votes": 0, "replies": [] }, { "id": 106134, "author_name": "", "author_url": "", "post_date": "2016-01-28T18:55:04.873000", "content": "", "votes": 0, "replies": [] }, { "id": 106086, "author_name": "", "author_url": "", "post_date": "2016-01-28T11:48:59.353000", "content": "", "votes": 0, "replies": [] }, { "id": 106075, "author_name": "", "author_url": "", "post_date": "2016-01-28T10:22:58.703000", "content": "", "votes": 0, "replies": [] }, { "id": 106038, "author_name": "", "author_url": "", "post_date": "2016-01-28T01:09:34.100000", "content": "", "votes": 0, "replies": [] }, { "id": 106026, "author_name": "", "author_url": "", "post_date": "2016-01-27T23:20:26.563000", "content": "", "votes": 0, "replies": [] }, { "id": 106000, "author_name": "", "author_url": "", "post_date": "2016-01-27T21:17:52.527000", "content": "", "votes": 0, "replies": [] }, { "id": 105991, "author_name": "", "author_url": "", "post_date": "2016-01-27T20:37:24.633000", "content": "", "votes": 0, "replies": [] }, { "id": 105985, "author_name": "", "author_url": "", "post_date": "2016-01-27T20:07:08.520000", "content": "", "votes": 0, "replies": [] }, { "id": 105982, "author_name": "", "author_url": "", "post_date": "2016-01-27T19:54:05.493000", "content": "", "votes": 0, "replies": [] }, { "id": 105981, "author_name": "", "author_url": "", "post_date": "2016-01-27T19:50:42.857000", "content": "", "votes": 0, "replies": [] }, { "id": 105978, "author_name": "", "author_url": "", "post_date": "2016-01-27T19:32:13.920000", "content": "", "votes": 0, "replies": [] }, { "id": 105968, "author_name": "", "author_url": "", "post_date": "2016-01-27T18:52:40.903000", "content": "", "votes": 0, "replies": [] }, { "id": 105963, "author_name": "", "author_url": "", "post_date": "2016-01-27T18:27:03.060000", "content": "", "votes": 0, "replies": [] }, { "id": 105943, "author_name": "", "author_url": "", "post_date": "2016-01-27T15:21:07.197000", "content": "", "votes": 0, "replies": [] }, { "id": 105942, "author_name": "", "author_url": "", "post_date": "2016-01-27T15:17:38.790000", "content": "", "votes": 0, "replies": [] }, { "id": 105873, "author_name": "", "author_url": "", "post_date": "2016-01-27T05:27:28.940000", "content": "", "votes": 0, "replies": [] }, { "id": 105815, "author_name": "", "author_url": "", "post_date": "2016-01-26T22:02:28.440000", "content": "", "votes": 0, "replies": [] }, { "id": 105797, "author_name": "", "author_url": "", "post_date": "2016-01-26T19:42:10.157000", "content": "", "votes": 0, "replies": [] }, { "id": 105794, "author_name": "", "author_url": "", "post_date": "2016-01-26T19:26:58.433000", "content": "", "votes": 0, "replies": [] }, { "id": 105678, "author_name": "", "author_url": "", "post_date": "2016-01-25T19:47:54.580000", "content": "", "votes": 0, "replies": [] }, { "id": 105677, "author_name": "", "author_url": "", "post_date": "2016-01-25T19:46:31.750000", "content": "", "votes": 0, "replies": [] }, { "id": 105670, "author_name": "", "author_url": "", "post_date": "2016-01-25T19:07:43.987000", "content": "", "votes": 0, "replies": [] }, { "id": 105667, "author_name": "", "author_url": "", "post_date": "2016-01-25T18:47:23.650000", "content": "", "votes": 0, "replies": [] }, { "id": 105653, "author_name": "", "author_url": "", "post_date": "2016-01-25T16:45:30.057000", "content": "", "votes": 0, "replies": [] }, { "id": 105650, "author_name": "", "author_url": "", "post_date": "2016-01-25T16:33:32.493000", "content": "", "votes": 0, "replies": [] }, { "id": 105648, "author_name": "", "author_url": "", "post_date": "2016-01-25T16:30:11.643000", "content": "", "votes": 0, "replies": [] }, { "id": 105636, "author_name": "", "author_url": "", "post_date": "2016-01-25T14:02:12.270000", "content": "", "votes": 0, "replies": [] }, { "id": 105629, "author_name": "", "author_url": "", "post_date": "2016-01-25T12:45:28.827000", "content": "", "votes": 0, "replies": [] }, { "id": 105621, "author_name": "", "author_url": "", "post_date": "2016-01-25T10:16:24.163000", "content": "", "votes": 0, "replies": [] }, { "id": 105546, "author_name": "", "author_url": "", "post_date": "2016-01-24T15:32:28.943000", "content": "", "votes": 0, "replies": [] }, { "id": 105505, "author_name": "", "author_url": "", "post_date": "2016-01-23T20:50:54.660000", "content": "", "votes": 0, "replies": [] }, { "id": 105504, "author_name": "", "author_url": "", "post_date": "2016-01-23T20:44:43.147000", "content": "", "votes": 0, "replies": [] }, { "id": 105503, "author_name": "", "author_url": "", "post_date": "2016-01-23T20:37:58.083000", "content": "", "votes": 0, "replies": [] }, { "id": 105493, "author_name": "", "author_url": "", "post_date": "2016-01-23T19:02:53.813000", "content": "", "votes": 0, "replies": [] }, { "id": 105488, "author_name": "", "author_url": "", "post_date": "2016-01-23T18:08:06.183000", "content": "", "votes": 0, "replies": [] }, { "id": 105487, "author_name": "", "author_url": "", "post_date": "2016-01-23T18:02:25.960000", "content": "", "votes": 0, "replies": [] }, { "id": 105485, "author_name": "", "author_url": "", "post_date": "2016-01-23T17:47:03.740000", "content": "", "votes": 0, "replies": [] }, { "id": 105478, "author_name": "", "author_url": "", "post_date": "2016-01-23T15:28:05.760000", "content": "", "votes": 0, "replies": [] }, { "id": 105476, "author_name": "", "author_url": "", "post_date": "2016-01-23T15:09:24.720000", "content": "", "votes": 0, "replies": [] }, { "id": 105472, "author_name": "", "author_url": "", "post_date": "2016-01-23T13:02:12.930000", "content": "", "votes": 0, "replies": [] }, { "id": 105454, "author_name": "", "author_url": "", "post_date": "2016-01-23T04:25:42.280000", "content": "", "votes": 0, "replies": [] }, { "id": 105401, "author_name": "", "author_url": "", "post_date": "2016-01-22T21:09:48.803000", "content": "", "votes": 0, "replies": [] }, { "id": 105165, "author_name": "", "author_url": "", "post_date": "2016-01-20T15:52:24.853000", "content": "", "votes": 0, "replies": [] }, { "id": 105161, "author_name": "", "author_url": "", "post_date": "2016-01-20T15:24:24.477000", "content": "", "votes": 0, "replies": [] }, { "id": 105155, "author_name": "", "author_url": "", "post_date": "2016-01-20T14:49:55.343000", "content": "", "votes": 0, "replies": [] }, { "id": 105154, "author_name": "", "author_url": "", "post_date": "2016-01-20T14:46:56.397000", "content": "", "votes": 0, "replies": [] }, { "id": 105140, "author_name": "", "author_url": "", "post_date": "2016-01-20T12:54:07.363000", "content": "", "votes": 0, "replies": [] }, { "id": 105139, "author_name": "", "author_url": "", "post_date": "2016-01-20T12:51:37.213000", "content": "", "votes": 0, "replies": [] }, { "id": 105078, "author_name": "", "author_url": "", "post_date": "2016-01-19T19:28:13.390000", "content": "", "votes": 0, "replies": [] }, { "id": 105074, "author_name": "", "author_url": "", "post_date": "2016-01-19T19:08:14.010000", "content": "", "votes": 0, "replies": [] }, { "id": 105058, "author_name": "", "author_url": "", "post_date": "2016-01-19T14:57:37.537000", "content": "", "votes": 0, "replies": [] }, { "id": 105053, "author_name": "", "author_url": "", "post_date": "2016-01-19T14:05:16.527000", "content": "", "votes": 0, "replies": [] }, { "id": 105052, "author_name": "", "author_url": "", "post_date": "2016-01-19T13:43:33.473000", "content": "", "votes": 0, "replies": [] }, { "id": 105022, "author_name": "", "author_url": "", "post_date": "2016-01-18T22:55:30.620000", "content": "", "votes": 0, "replies": [] }, { "id": 105016, "author_name": "", "author_url": "", "post_date": "2016-01-18T21:31:32.240000", "content": "", "votes": 0, "replies": [] }, { "id": 104986, "author_name": "", "author_url": "", "post_date": "2016-01-18T16:47:40.843000", "content": "", "votes": 0, "replies": [] }, { "id": 104981, "author_name": "", "author_url": "", "post_date": "2016-01-18T15:18:59.287000", "content": "", "votes": 0, "replies": [] }, { "id": 104978, "author_name": "", "author_url": "", "post_date": "2016-01-18T15:00:19.440000", "content": "", "votes": 0, "replies": [] }, { "id": 104973, "author_name": "", "author_url": "", "post_date": "2016-01-18T14:12:09.157000", "content": "", "votes": 0, "replies": [] }, { "id": 104936, "author_name": "", "author_url": "", "post_date": "2016-01-18T03:42:51.157000", "content": "", "votes": 0, "replies": [] }, { "id": 104933, "author_name": "", "author_url": "", "post_date": "2016-01-18T03:32:23.480000", "content": "", "votes": 0, "replies": [] }, { "id": 104930, "author_name": "", "author_url": "", "post_date": "2016-01-18T03:04:44.057000", "content": "", "votes": 0, "replies": [] }, { "id": 104929, "author_name": "", "author_url": "", "post_date": "2016-01-18T03:04:34.690000", "content": "", "votes": 0, "replies": [] }, { "id": 104906, "author_name": "", "author_url": "", "post_date": "2016-01-17T20:38:43.173000", "content": "", "votes": 0, "replies": [] }, { "id": 104883, "author_name": "", "author_url": "", "post_date": "2016-01-17T18:18:08.407000", "content": "", "votes": 0, "replies": [] }, { "id": 104867, "author_name": "", "author_url": "", "post_date": "2016-01-17T15:17:18.793000", "content": "", "votes": 0, "replies": [] }, { "id": 104827, "author_name": "", "author_url": "", "post_date": "2016-01-17T02:21:49.657000", "content": "", "votes": 0, "replies": [] }, { "id": 104744, "author_name": "", "author_url": "", "post_date": "2016-01-16T05:57:11.787000", "content": "", "votes": 0, "replies": [] }, { "id": 104743, "author_name": "", "author_url": "", "post_date": "2016-01-16T05:50:29.627000", "content": "", "votes": 0, "replies": [] }, { "id": 104428, "author_name": "", "author_url": "", "post_date": "2016-01-12T16:42:35.197000", "content": "", "votes": 0, "replies": [] }, { "id": 104352, "author_name": "", "author_url": "", "post_date": "2016-01-12T02:02:02.967000", "content": "", "votes": 0, "replies": [] }, { "id": 104248, "author_name": "", "author_url": "", "post_date": "2016-01-10T23:48:37.280000", "content": "", "votes": 0, "replies": [] }, { "id": 104246, "author_name": "", "author_url": "", "post_date": "2016-01-10T23:22:39.373000", "content": "", "votes": 0, "replies": [] }, { "id": 104242, "author_name": "", "author_url": "", "post_date": "2016-01-10T22:05:24.943000", "content": "", "votes": 0, "replies": [] }, { "id": 104241, "author_name": "", "author_url": "", "post_date": "2016-01-10T22:03:21.350000", "content": "", "votes": 0, "replies": [] }, { "id": 104229, "author_name": "", "author_url": "", "post_date": "2016-01-10T18:12:09.043000", "content": "", "votes": 0, "replies": [] }, { "id": 104228, "author_name": "", "author_url": "", "post_date": "2016-01-10T18:11:44.787000", "content": "", "votes": 0, "replies": [] }, { "id": 104227, "author_name": "", "author_url": "", "post_date": "2016-01-10T18:04:03.040000", "content": "", "votes": 0, "replies": [] }, { "id": 104226, "author_name": "", "author_url": "", "post_date": "2016-01-10T17:52:47.180000", "content": "", "votes": 0, "replies": [] }, { "id": 104223, "author_name": "", "author_url": "", "post_date": "2016-01-10T17:15:00.083000", "content": "", "votes": 0, "replies": [] }, { "id": 104222, "author_name": "", "author_url": "", "post_date": "2016-01-10T17:11:12.097000", "content": "", "votes": 0, "replies": [] }, { "id": 104218, "author_name": "", "author_url": "", "post_date": "2016-01-10T15:46:46.027000", "content": "", "votes": 0, "replies": [] }, { "id": 104166, "author_name": "", "author_url": "", "post_date": "2016-01-09T21:17:26.530000", "content": "", "votes": 0, "replies": [] }, { "id": 104164, "author_name": "", "author_url": "", "post_date": "2016-01-09T21:06:13.190000", "content": "", "votes": 0, "replies": [] }, { "id": 104163, "author_name": "", "author_url": "", "post_date": "2016-01-09T21:02:18.677000", "content": "", "votes": 0, "replies": [] }, { "id": 104150, "author_name": "", "author_url": "", "post_date": "2016-01-09T17:26:45.663000", "content": "", "votes": 0, "replies": [] }, { "id": 104148, "author_name": "", "author_url": "", "post_date": "2016-01-09T17:19:47.970000", "content": "", "votes": 0, "replies": [] }, { "id": 104111, "author_name": "", "author_url": "", "post_date": "2016-01-09T04:57:09.143000", "content": "", "votes": 0, "replies": [] }, { "id": 104110, "author_name": "", "author_url": "", "post_date": "2016-01-09T04:51:44.557000", "content": "", "votes": 0, "replies": [] }, { "id": 104090, "author_name": "", "author_url": "", "post_date": "2016-01-09T00:52:42.010000", "content": "", "votes": 0, "replies": [] }, { "id": 104064, "author_name": "", "author_url": "", "post_date": "2016-01-08T21:18:56.637000", "content": "", "votes": 0, "replies": [] }, { "id": 104014, "author_name": "", "author_url": "", "post_date": "2016-01-08T15:13:19.617000", "content": "", "votes": 0, "replies": [] }, { "id": 103899, "author_name": "", "author_url": "", "post_date": "2016-01-07T19:15:13.707000", "content": "", "votes": 0, "replies": [] }, { "id": 103897, "author_name": "", "author_url": "", "post_date": "2016-01-07T17:09:05.867000", "content": "", "votes": 0, "replies": [] }, { "id": 103801, "author_name": "", "author_url": "", "post_date": "2016-01-06T16:09:04.820000", "content": "", "votes": 0, "replies": [] }, { "id": 103546, "author_name": "", "author_url": "", "post_date": "2016-01-04T07:47:24.527000", "content": "", "votes": 0, "replies": [] }, { "id": 103529, "author_name": "", "author_url": "", "post_date": "2016-01-04T01:21:31.967000", "content": "", "votes": 0, "replies": [] }, { "id": 103479, "author_name": "", "author_url": "", "post_date": "2016-01-03T15:37:54.727000", "content": "", "votes": 0, "replies": [] }, { "id": 103406, "author_name": "", "author_url": "", "post_date": "2016-01-02T10:36:14.993000", "content": "", "votes": 0, "replies": [] }, { "id": 103405, "author_name": "", "author_url": "", "post_date": "2016-01-02T10:21:55.063000", "content": "", "votes": 0, "replies": [] }, { "id": 103388, "author_name": "", "author_url": "", "post_date": "2016-01-02T02:32:00.130000", "content": "", "votes": 0, "replies": [] }, { "id": 103347, "author_name": "", "author_url": "", "post_date": "2016-01-01T11:29:34.913000", "content": "", "votes": 0, "replies": [] }, { "id": 103323, "author_name": "", "author_url": "", "post_date": "2015-12-31T20:03:51.107000", "content": "", "votes": 0, "replies": [] }, { "id": 103310, "author_name": "", "author_url": "", "post_date": "2015-12-31T17:07:57.397000", "content": "", "votes": 0, "replies": [] }, { "id": 103307, "author_name": "", "author_url": "", "post_date": "2015-12-31T16:53:33.303000", "content": "", "votes": 0, "replies": [] }, { "id": 103255, "author_name": "", "author_url": "", "post_date": "2015-12-30T19:25:28.893000", "content": "", "votes": 0, "replies": [] }, { "id": 103101, "author_name": "", "author_url": "", "post_date": "2015-12-28T23:06:42.127000", "content": "", "votes": 0, "replies": [] }, { "id": 103100, "author_name": "", "author_url": "", "post_date": "2015-12-28T22:38:48.007000", "content": "", "votes": 0, "replies": [] }, { "id": 103059, "author_name": "", "author_url": "", "post_date": "2015-12-28T17:43:51.710000", "content": "", "votes": 0, "replies": [] }, { "id": 103041, "author_name": "", "author_url": "", "post_date": "2015-12-28T14:56:58.077000", "content": "", "votes": 0, "replies": [] }, { "id": 103017, "author_name": "", "author_url": "", "post_date": "2015-12-28T12:09:46.417000", "content": "", "votes": 0, "replies": [] }, { "id": 102970, "author_name": "", "author_url": "", "post_date": "2015-12-28T05:33:20.010000", "content": "", "votes": 0, "replies": [] }, { "id": 102932, "author_name": "", "author_url": "", "post_date": "2015-12-27T21:49:14.630000", "content": "", "votes": 0, "replies": [] }, { "id": 102917, "author_name": "", "author_url": "", "post_date": "2015-12-27T18:43:51.260000", "content": "", "votes": 0, "replies": [] }, { "id": 102913, "author_name": "", "author_url": "", "post_date": "2015-12-27T17:54:12.737000", "content": "", "votes": 0, "replies": [] }, { "id": 102909, "author_name": "", "author_url": "", "post_date": "2015-12-27T17:46:53.210000", "content": "", "votes": 0, "replies": [] }, { "id": 102908, "author_name": "", "author_url": "", "post_date": "2015-12-27T17:45:09.167000", "content": "", "votes": 0, "replies": [] }, { "id": 102877, "author_name": "", "author_url": "", "post_date": "2015-12-27T09:06:11.407000", "content": "", "votes": 0, "replies": [] }, { "id": 102873, "author_name": "", "author_url": "", "post_date": "2015-12-27T07:25:21.147000", "content": "", "votes": 0, "replies": [] }, { "id": 102691, "author_name": "", "author_url": "", "post_date": "2015-12-24T13:39:05.923000", "content": "", "votes": 0, "replies": [] }, { "id": 102660, "author_name": "", "author_url": "", "post_date": "2015-12-24T08:04:34.337000", "content": "", "votes": 0, "replies": [] }, { "id": 102655, "author_name": "", "author_url": "", "post_date": "2015-12-24T07:28:26.300000", "content": "", "votes": 0, "replies": [] }, { "id": 102601, "author_name": "", "author_url": "", "post_date": "2015-12-23T20:23:34.850000", "content": "", "votes": 0, "replies": [] }, { "id": 102599, "author_name": "", "author_url": "", "post_date": "2015-12-23T20:07:58.453000", "content": "", "votes": 0, "replies": [] }, { "id": 102534, "author_name": "", "author_url": "", "post_date": "2015-12-23T13:00:08.637000", "content": "", "votes": 0, "replies": [] }, { "id": 102513, "author_name": "", "author_url": "", "post_date": "2015-12-23T07:41:21.520000", "content": "", "votes": 0, "replies": [] }, { "id": 102508, "author_name": "", "author_url": "", "post_date": "2015-12-23T06:21:04.070000", "content": "", "votes": 0, "replies": [] }, { "id": 102437, "author_name": "", "author_url": "", "post_date": "2015-12-22T17:05:22.937000", "content": "", "votes": 0, "replies": [] }, { "id": 104478, "author_name": "", "author_url": "", "post_date": "2016-01-13T04:27:23.470000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "102412": "Hi All,\r\n\r\nWe just make an end-to-end deep learning tutorial with lb score 0.0392:\r\n\r\nhttps://github.com/dmlc/mxnet/tree/master/example/kaggle-ndsb2\r\n\r\n\r\nNotice this is a very simple model with no attempt to optimize the structure or hyper parameters, you can build fantastic network based on it. While this tutorial is written in python, MXNet comes with support for other popular languages such as R and Julia which can also be used. You are more than welcomed to try and contribute back to this example.\r\n\r\nBests,\r\n\r\nBing", "103814": "@EIGSI, I don't want to give away too many secrets, but here are some general suggestions:\r\n\r\n1. With validation curves like these, you should definitely look into early stopping if you aren't using it already! You appear to be overfitting pretty seriously.\r\n2. Look into data augmentation. Did I mention that you seem to be overfitting ;-)\r\n3. Look at ways to improve the preprocessing code. \r\n\r\nNumbers 1 and 2 are pretty standard for neural nets and I nearly always use them. Number 3 is always worth trying, it's fairly quick to try and can sometimes have a large payoff. \r\n\r\nI wouldn't worry about learning rate or momentum tuning till you fix your overfitting issues. All you would likely be able to do is to increase the amount of overfit. Averaging the results of cross validation is a perfectly reasonable idea, but likely very, very slow. It's not something I'd try till I'd completely run out of other ideas.\r\n\r\nGood luck!\r\n", "102448": "thumbs up for MXnet!\r\n\r\nFor people who want to install GPU version of MXnet, please refer to this blog https://no2147483647.wordpress.com/2015/12/07/deep-learning-for-hackers-with-mxnet-1/\r\n\r\nPlus a fun of neural art with MXnet: https://no2147483647.wordpress.com/2015/12/21/deep-learning-for-hackers-with-mxnet-2/", "103815": "@WD you are welcome. Yes, AWS GPU instance needs some kernel image update for installing CUDA if ones wants to install from Ubuntu scratch (https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN) ) I will make a MXnet version for it. Or, one can use docker anyway http://mxnt.ml/en/latest/build.html#docker-images\r\n\r\nAnd, 'star-ed' means 'click the star button on the right top and add a star to MXnet github' :-) I am wondering why the slow tensorflow has 10k+ stars on github while all core code pull requests being rejected (you call it open source?), but the faster, easy-to-use, real open source MXnet only has 2000+ stars now. Please click the star button https://github.com/dmlc/mxnet \r\n\r\nThe reasons why MXnet is good for competitions:\r\n\r\n1. it is faster and memory efficient: Alex Smola's NIPS 2015 talk http://alex.smola.org/talks/NIPS15.pdf page 52 had speed comparison for MXnet vs tensorflow/torch/theano/caffe. MXnet won in most cases for memory and had about the same speed as others except tensorflow: MXnet was much faster than tensorflow .\r\n2. It natively supports multiple language: python/R/Java/Julia/C++ and java script, so \r\n3. It supports multi-GPU and distributed training while tensorflow doesn't support distributed training (well, the open source one doesn't). The winning solution for this competition may use 20+ machines where 4-8 GPUs each :-)\r\n4. The only (useful) deep learning tutorial in this competition uses MXnet (Python/R) :-)", "103812": "@EIGSI some of my tricks/techniques:\r\n\r\n1. data augment is necessary, you can check out the first data science bowl winning solutions and find the mirror/rotation/strength/resize etc\r\n2. longer training: epoch*batch_size*learning_rate\r\n3. deeper ConvNet: the current tutorial of LeNet is too shallow. A winning solution for this competition expects 15+ layers, I guess.\r\n4. Rent an Amazon AWS: all the techniques above will soon burn your macbook pro video card. Installing Mxnet on AWS may need some tricks, I will make a quick and no pain tutorial for it.\r\n5. Always use MXnet. Have you star-ed MXnet github repo https://github.com/dmlc/mxnet ? Click the 'star' button and get some luck in training.", "102433": "Thank you so much for sharing (and advancing mxnet)!\r\nI had a small bug which is in the line \r\n\r\n img = preproc(f.pixel_array / np.max(f.pixel_array))\r\n\r\nThe problem is that for me, `dicom`, returns int pixels so dividing by the max returns an all zero image (except for the max) so a possible fix is:\r\n\r\n img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\r\n", "105656": "The easiest should use SFrame.\r\nI will write a tutorial on how to use SFrame with MXNet. Hopefully in this week.\r\nhttps://github.com/dmlc/mxnet/tree/master/plugin/sframe\r\n\r\nhttps://github.com/dato-code/SFrame\r\n\r\n\r\n[quote=WD;105654]\r\n\r\n@phunter. thanks for this. i will look into both the CSV route and the Io.record route. in the CSV approach - i guess the easiest way would be to work with the numpy \"img\" object - no? i will play around with this and make code public if i find something useful to others\r\n\r\n f = dicom.read_file(path)\r\n img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\r\n dst_path = path.rsplit(\".\", 1)[0] + \".64x64.jpg\" #create jpg out of all files\r\n\r\n[/quote]", "106773": "[quote=WD;106742]\r\n\r\n@Franc. thanks so much. should one add as well a convolutional relu layer between the flatten and the fc1? as per below? or is that not correct / needed? \r\n\r\n flatten = mx.symbol.Flatten(net)\r\n flatten = mx.symbol.Dropout(flatten)\r\n flatten = mx.sym.Activation(flatten, act_type=\"relu\") ###to be checked\r\n fc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=1024)\r\n fc1 = mx.sym.Activation(fc1, act_type=\"relu\")\r\n fc1 = mx.symbol.Dropout(fc1)\r\n fc2 = mx.symbol.FullyConnected(data=fc1, num_hidden=1)\r\n\r\n\r\n[/quote]\r\nNo, usually you do not put convolutinal layer between flatten and FC layer. Notice, that in the third line in the code above you have only activation layer. I would simply remove the third line of code, i.e. `flatten = mx.sym.Activation(flatten, act_type=\"relu\")`", "106664": "I just want to stick this out here for those still stuck trying to process on Windows with Python 3.5. The issue with the processing file failing at on the last line is because when using csv.writer on a Windows machine it will add a extra line to each row of data. To fix this error open the file that you want to write to with open(file, \"w\", newline = ''). Many thanks to Scott Smith for telling me this almost of month ago. Sad thing is I just realized it today after I fixed the code. ", "106343": "@Linear regression. Tooled the Mxnet tutoral over from logistic to linear regression. happy to share code if of interest to others ", "106251": "@Tim. At a more elementary level, i was thinking, that if the network outputs (on e.g. a hypothetical 8 output notes, labelled from 1 to 8): 0.1, 0.1, 0.1, 0.1, 0.99, 0.99, 0.99, 0.99.\r\n\r\nThen the Mxnet would create a CDF that would be 0.1, 0.1, 0.1, 0.1, 0.99, 0.99, 0.99, 0.99. This is almost a step function at node 5. \r\n\r\nHowever, as the network is equally confident that the output might be 5,6,7 and 8, a naieve best guess on best estimator would potentially be the average between 5 to 8, e.g. 6.5 \r\n\r\n(normalizing the inputs and subsequently calculating a CDF of e.g. (0.025,0.05,0.075,0.1,0.325,0.55,0.775,1) also does intuitively not make sense) \r\n\r\nHence - I was wondering whether there is an opportunity / need to adjust the CDF that the Mxnet tutoral calculates? I realize that this is a trivial example - but still would love to get your thoughts. \r\n", "106079": "@Bartek that's the beauty of neural networks they capture the \"je ne sais quoi\" (I DONT KNOW WHAT) in the data :). What makes you able to distinguish faces of 2 different people, you can't really explain why? There are some obvious things you see but there are latent features baked into your brain that make you able to distinguish faces. That's what neural networks capture. The technical explanation is that they capture hierarchical representations of features in the data. In my opinion the first tutorial is trying to explain why 2 faces are different (segmenting e.t.c) there are just too many variables, scale, noise, orientation e.t.c for it to do a good job. Early in the competition people who tried that approach reported scores of 0.1xxxx which is very bad compared to the current scores on the leaderboard.", "106024": "@WD, that's an interesting point. If my quick scribblings are correct, then computing the MSE on the CDF (versus a step function) is equivalent to computing a weighted MSE on the PDF. The weights would be linearly decreasing[1]. So, for instance the weights could be 600, 599, ...., 2, 1. Thus, if one is outputting the CDF directly one is putting more emphasis on the lower values than upper values. This makes intuitive sense, since an incorrect early value in the PDF will result in all subsequent values of the CDF being off. \r\n\r\nI would suspect that computing the CDF directly will result in better scores than computing the PDF (using softmax) and then converting that to CDF since CDF is the eventual target. I may be wrong about that though, so I encourage you to try it. \r\n\r\n[EDIT] OK, I looked at it a bit more and that math is wrong, but I think the conclusion is probably more or less right, earlier points end up being weighted more than later points.\r\n\r\n\r\n[1] Whether the weights are linear or quadratic depends on whether one is applying them before or after squaring. Here I assume that they are applied before squaring.", "106009": "@Tim. As a follow-up thought - it seems that the indepedent sigmoid approach and the resulting CDF approximation creates a CDF that is a little \"biased to the left\". A actual softmax \"adjustment\" and proper CDF calculation from these values might lead to an improvement in scores. will try this out later. ", "105964": "\r\n@WD, even though this layer is *labelled* \"softmax\", I believe it's just a sigmoid output. The net is supposed to figure out that the output should look like a CDF during training, so the output should be approximately ascending, but it can have glitches. That is why there is `submission_helper`, it fixes up any non-monotonicity.\r\n\r\nHope that helps.", "105654": "@phunter. thanks for this. i will look into both the CSV route and the Io.record route. in the CSV approach - i guess the easiest way would be to work with the numpy \"img\" object - no? i will play around with this and make code public if i find something useful to others\r\n\r\n f = dicom.read_file(path)\r\n img = preproc(f.pixel_array.astype(float) / np.max(f.pixel_array))\r\n dst_path = path.rsplit(\".\", 1)[0] + \".64x64.jpg\" #create jpg out of all files", "105651": "There is rotation and a bunch of augmentation method, but require ImageRecord IO. For this example, we only want to show how to use MXNet so we choose CSV.\r\n\r\nhttps://mxnet.readthedocs.org/en/latest/python/io.html#mxnet.io.ImageRecordIter\r\n\r\n\r\n[quote=phunter;105650]\r\n\r\nthe mxnet tutorial code has not (yet) support image rotation. one can modify preprocessing.py and generate CSV with rotation by scikit-image\r\n\r\n[/quote]", "105164": "G2x2 spot runs at around 0.1$ per hour. Regular instances (not spot) are more expensive - 0.65$. it depends on your launch zone. The price graphs can be found on AWS - when go to spot instances.\r\n", "105066": "I am finishing up code to extract and plot training and validation curves, and I will share this code today or tomorrow. I had a quick question. If one uses a terminal mode to access one's AWS instance (e.g. using Putty) and either runs scripts from the bash command line or from an ipython shell, then is it possible to plot graphs visually? or does one either a) have to install an ipython notebook, or b) save-to-image, transfer the files back to one's local computer, and plot locally? ", "105013": "i am trying to visualize the network. Some others might be interested in this as well. I thought the code below should do the trick. However, even doing all the sudo apt-get installs (ubuntu) i still get a graphviz library not found error. let me know if anyone has experience with this or has encountered the same issue or knows how to resolve this! \r\n\r\n\r\n> import graphviz import find_mxnet import mxnet as mx import importlib\r\n> \r\n> def get_lenet():\r\n> #the code from the train.py file \r\n\r\n> network = get_lenet()\r\n> \r\n> mx.viz.plot_network(network)\r\n\r\n", "104979": "@EIGSI - you can also log the validation curves by using e.g. stytole_model.fit(X=data_train, eval_metric = mx.metric.np(CRPS),batch_end_callback = mx.callback.log_train_metric(32))\r\n", "104955": "@Jon Carlies: since precompiled mxnet libraries are only cpu enabled, replace all : `devs = [mx.gpu(0)]` with `devs = [mx.cpu()]`. ", "104904": "[quote=Scott Smith;104308]\r\n\r\n[quote=Matthew Tubs;104174]\r\n\r\nUpdate: I got the code running smoother now. Turned out that while messing with R I made images of all the dicom files and that was interrupting the code. \r\n\r\nHowever, I have ran into a different error at the end of the run. Can anyone help me figure why this index went out of range?\r\n\r\n if split_to_train[cnt]:\r\n\r\nIndexError: list index out of range\r\n\r\n[/quote]\r\n\r\n@Matthew: I think I ran into the same issues preprocessing with Python 3.5 on Windows.\r\n\r\n 1. In the get_frames() function, the second time you run there are no frames detected due to the presence of jpg images introduced. The adjustment in [@Woolsey's comment][1] corrects this by limiting the 'files' object to only dcm entries.\r\n 2. With Python 3 and Windows, there can be line returns introduced where you don't want them. Check your train-label.csv file. I had additional lines introduced in the get_label_map() function that would lead to empty rows in train_label.csv. I think I fixed this by using 'fi = open(fname, 'r', newline ='')' in get_label_map(), but the file is not on this computer. It might help to try preprocessing with the first 5 train and validation studies only and then take a look at the csv's. If there are extra lines or empty lines at the end of the files, this will be a problem when you process your larger set.\r\n\r\n [1]: https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/18079/end-to-end-deep-learning-tutorial-0-0392/103854#post103854\r\n\r\n[/quote]\r\n\r\nI'm on the last line of code and reviewing #2 in this post to try and solve my index issue. For me, he train_frames has a length of 5,293 yet train-64x64-data.csv has double that which I believe is why there is an index error. Am I understanding this correctly...how many rows should I have in train-64x64-data.csv?\r\n", "104308": "[quote=Matthew Tubs;104174]\r\n\r\nUpdate: I got the code running smoother now. Turned out that while messing with R I made images of all the dicom files and that was interrupting the code. \r\n\r\nHowever, I have ran into a different error at the end of the run. Can anyone help me figure why this index went out of range?\r\n\r\n if split_to_train[cnt]:\r\n\r\nIndexError: list index out of range\r\n\r\n[/quote]\r\n\r\n@Matthew: I think I ran into the same issues preprocessing with Python 3.5 on Windows.\r\n\r\n 1. In the get_frames() function, the second time you run there are no frames detected due to the presence of jpg images introduced. The adjustment in [@Woolsey's comment][1] corrects this by limiting the 'files' object to only dcm entries.\r\n 2. With Python 3 and Windows, there can be line returns introduced where you don't want them. Check your train-label.csv file. I had additional lines introduced in the get_label_map() function that would lead to empty rows in train_label.csv. I think I fixed this by using 'fi = open(fname, 'r', newline ='')' in get_label_map(), but the file is not on this computer. It might help to try preprocessing with the first 5 train and validation studies only and then take a look at the csv's. If there are extra lines or empty lines at the end of the files, this will be a problem when you process your larger set.\r\n\r\n [1]: https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/18079/end-to-end-deep-learning-tutorial-0-0392/103854#post103854", "104174": "[quote=Matthew Tubs;104166]\r\n\r\n[quote=earino;104164]\r\n\r\n[quote=Matthew Tubs;104163]\r\nFranc \r\n\r\nI've been trying to get the Preprocessing.py file you attached running and I've ran into a issue. Would you help me out in understanding what is going on? I am running Python 3.5 on windows, and I am receiving the following error.\r\n\r\n tran_index = np.loadtxt(\"./tran_label.csv\", delimiter=\",\")[:,0].astype(\"int\")\r\n\r\nIndexError: too many indices for array\r\n\r\nThanks much\r\n [/quote]\r\n\r\nis that tran_label supposed to be train_label?\r\n\r\n\r\n[/quote]\r\nI did a replace all to see if the problem was something attached to the name of the file.\r\n\r\n[/quote]\r\n\r\nUpdate: I got the code running smoother now. Turned out that while messing with R I made images of all the dicom files and that was interrupting the code. \r\n\r\nHowever, I have ran into a different error at the end of the run. Can anyone help me figure why this index went out of range?\r\n\r\n if split_to_train[cnt]:\r\n\r\nIndexError: list index out of range", "104116": "Hi! What is the license of your tutorial? I don't see one posted in the github?", "104026": "i received an odd error running processing.py. what might be the root-cause? \r\n\r\nW\r\n\r\n800 slices processed\r\nTraceback (most recent call last):\r\n File \"Preprocessing.py\", line 134, in