Hi, all,
\n\nWe are providing a deep learning model in R using the mxnet package.
https://github.com/dmlc/mxnet/blob/master/example/kaggle-ndsb2/Train.R
\n\nIt is the same network structure and parameters with the model from Bing Xu. And of course they have the same lb score.
\n\nAs a R user for years, I personally feel a little unhappy that we don't have good deep learning tools in R. Even the deep learning tutorials on Kaggle are all using python.
\n\nNow we have mxnet!
To install the mxnet R package, please follow the instructions from:
http://mxnet.readthedocs.org/en/latest/build.html#r-package-installation
\n\nWe also provide several tutorials:
\n\nhttp://mxnet.readthedocs.org/en/latest/R-package/index.html#tutorials
\n\nRight now the model is using pre-processed csv files by python code. I will add the preprocessing R code later.
", "messages": [ { "id": "102737", "postDate": "12/25/2015 05:30:05", "content": "Hi, all,
\n\nWe are providing a deep learning model in R using the mxnet package.
https://github.com/dmlc/mxnet/blob/master/example/kaggle-ndsb2/Train.R
\n\nIt is the same network structure and parameters with the model from Bing Xu. And of course they have the same lb score.
\n\nAs a R user for years, I personally feel a little unhappy that we don't have good deep learning tools in R. Even the deep learning tutorials on Kaggle are all using python.
\n\nNow we have mxnet!
To install the mxnet R package, please follow the instructions from:
http://mxnet.readthedocs.org/en/latest/build.html#r-package-installation
\n\nWe also provide several tutorials:
\n\nhttp://mxnet.readthedocs.org/en/latest/R-package/index.html#tutorials
\n\nRight now the model is using pre-processed csv files by python code. I will add the preprocessing R code later.
", "rawMarkdown": "Hi, all, \r\n\r\nWe are providing a deep learning model in R using the `mxnet` package. \r\n\r\nhttps://github.com/dmlc/mxnet/blob/master/example/kaggle-ndsb2/Train.R\r\n\r\nIt is the same network structure and parameters with the model from Bing Xu. And of course they have the same lb score.\r\n\r\nAs a R user for years, I personally feel a little unhappy that we don't have good deep learning tools in R. Even the deep learning tutorials on Kaggle are all using python.\r\n\r\nNow we have `mxnet`! \r\n\r\nTo install the `mxnet` R package, please follow the instructions from:\r\n\r\nhttp://mxnet.readthedocs.org/en/latest/build.html#r-package-installation\r\n\r\nWe also provide several tutorials:\r\n\r\nhttp://mxnet.readthedocs.org/en/latest/R-package/index.html#tutorials\r\n\r\nRight now the model is using pre-processed csv files by python code. I will add the preprocessing R code later.", "votes": null }, { "id": "102745", "postDate": "12/25/2015 09:20:20", "content": "Thumbs up for MxNet team! Great job.
\n\nOne probem appeared to me. When using the code I get the following errors:
\n\n> source <- (source-128) / 128\nError in source - 128: non-numeric argument to a binary operator\nand
\n\n> network <- get.lenet()\nError in frames[[i + 1]] : wrong arguments for subsetting an environment\nHi guys! I am trying to install the mxnet package, and it fails with the following:
\n\n\n\n\ndrat:::addRepo("dmlc")\n install.packages("mxnet")\n Warning in install.packages :\n cannot open URL 'http://dmlc.github.io/drat/src/contrib/PACKAGES.gz': HTTP status was '404 Not Found'\n Warning in install.packages :\n cannot open URL 'http://dmlc.github.io/drat/src/contrib/PACKAGES': HTTP status was '404 Not Found'\n Warning in install.packages :\n unable to access index for repository http://dmlc.github.io/drat/src/contrib:\n cannot open URL 'http://dmlc.github.io/drat/src/contrib/PACKAGES'
\n
I have tried to also access the URLs above, and they don't work. Any thoughts?
\n\nMany thanks!
", "rawMarkdown": "Hi guys! I am trying to install the mxnet package, and it fails with the following:\r\n\r\n> drat:::addRepo(\"dmlc\")\r\n> install.packages(\"mxnet\")\r\nWarning in install.packages :\r\n cannot open URL 'http://dmlc.github.io/drat/src/contrib/PACKAGES.gz': HTTP status was '404 Not Found'\r\nWarning in install.packages :\r\n cannot open URL 'http://dmlc.github.io/drat/src/contrib/PACKAGES': HTTP status was '404 Not Found'\r\nWarning in install.packages :\r\n unable to access index for repository http://dmlc.github.io/drat/src/contrib:\r\n cannot open URL 'http://dmlc.github.io/drat/src/contrib/PACKAGES'\r\n\r\nI have tried to also access the URLs above, and they don't work. Any thoughts?\r\n\r\nMany thanks!", "votes": null }, { "id": "102751", "postDate": "12/25/2015 13:58:32", "content": "I have done the following and it works for me:
\n\ndrat package from CRANE (I have done this in RStudio: Tools / Install Packages)drat:::addRepo("dmlc") in R Shellinstall.packages("mxnet") in R shellI have done the following and it works for me:
\n\ndrat package from CRANE (I have done this in RStudio: Tools / Install Packages)\nWhen I used install.packages("drat", repos="https://cran.rstudio.com") in R Shell, I've got simmilar errordrat:::addRepo("dmlc") in R Shellinstall.packages("mxnet") in R shellThanks a lot, Franc! It works in this way. I am also getting the errors that you described above.\nFor the 1st one, I did something like this and got past it (not sure if it's correct though):\n source <- mx.symbol.Variable("data")\n source2 <- mx.symbol.Variable("128")\n source <- (source-source2) / source2
\n\nHowever, I did not find any way around the 2nd one and moreover this one:\n Error: could not find function "mx.io.CSVIter". The python MxNet version also has this issue.\nThat is because the pre-built Windows package is behind the master from the repo.\nI don't have cuDNN yet (waiting for approval of NVidia user) so I guess I will wait for the next pre-built package for both Python & R. Hope the MxNet team will upload it soon.
", "rawMarkdown": "Thanks a lot, Franc! It works in this way. I am also getting the errors that you described above.\r\nFor the 1st one, I did something like this and got past it (not sure if it's correct though):\r\n source <- mx.symbol.Variable(\"data\")\r\n source2 <- mx.symbol.Variable(\"128\")\r\n source <- (source-source2) / source2\r\n\r\nHowever, I did not find any way around the 2nd one and moreover this one:\r\n Error: could not find function \"mx.io.CSVIter\". The python MxNet version also has this issue.\r\nThat is because the pre-built Windows package is behind the master from the repo.\r\nI don't have cuDNN yet (waiting for approval of NVidia user) so I guess I will wait for the next pre-built package for both Python & R. Hope the MxNet team will upload it soon.", "votes": null }, { "id": "102757", "postDate": "12/25/2015 16:35:11", "content": "Hi, guys,
\n\nSorry for being a little late. BlackCore is correct, the prebuilt packages are updated weekly, so it might be behind the github master.
\n\nI have just updated the Mac and Windows package. You should be able to install it.
\n\nNote: all prebuilt packages are CPU-only. If you want to use your GPU or you are a Linux hacker, you should compile it by yourself.
", "rawMarkdown": "Hi, guys,\r\n\r\nSorry for being a little late. BlackCore is correct, the prebuilt packages are updated weekly, so it might be behind the github master.\r\n\r\nI have just updated the Mac and Windows package. You should be able to install it.\r\n\r\nNote: all prebuilt packages are CPU-only. If you want to use your GPU or you are a Linux hacker, you should compile it by yourself.", "votes": null }, { "id": "102759", "postDate": "12/25/2015 16:51:29", "content": "Thanks Qiang. Now the above errors have gone. :)
\n\nThere is a new error that I have faced:
\n\n> net <- mx.symbol.BatchNorm(net, fix.gamma = TRUE)\nError: Cannot find argument 'fix_gamma', Possible Arguments:\n----------------\neps : float, optional, default=1e-010\n Epsilon to prevent div 0\nmomentum : float, optional, default=0.9\n Momentum for moving average\nHowever, when I use:
\n\n net <- mx.symbol.BatchNorm(data=net, eps=0.001, momentum=0.9 )\nthere is no error returned by this function. I suspect that the TRUE is not appropriate parameter for fix.gamma.
The next error I have faced and I do not know, what is the possible reason:
\n\n> data_train <- mx.io.CSVIter(\n+ data.csv = "./train-64x64-data.csv", data.shape = c(64, 64, 30),\n+ label.csv = "./train-stytole.csv", label.shape = 600,\n+ batch.size = batch_size\n+ )\nError in mx.io.CSVIter(data.csv = "./train-64x64-data.csv", data.shape = c(64, : \n could not find function "mx.varg.io.CSVIter"\nI think both of errors you met are due to the Windows lib is behind github master. Similar situations in python, since both of R and python are using the same Windows lib file.
\n\nPlease wait for some time.
", "rawMarkdown": "I think both of errors you met are due to the Windows lib is behind github master. Similar situations in python, since both of R and python are using the same Windows lib file.\r\n\r\nPlease wait for some time.", "votes": null }, { "id": "102783", "postDate": "12/25/2015 22:50:41", "content": "Thanks.
", "rawMarkdown": "Thanks.", "votes": null }, { "id": "102803", "postDate": "12/26/2015 06:45:57", "content": "mxnet rocks with R!
", "rawMarkdown": "mxnet rocks with R!", "votes": null }, { "id": "102868", "postDate": "12/27/2015 06:55:24", "content": "@Qiang: When do you plan to release a weekly update of windows package?
", "rawMarkdown": "Qiang: When do you plan to release a weekly update of windows package?", "votes": null }, { "id": "102869", "postDate": "12/27/2015 06:55:41", "content": "Did any one try to implement in-memory solution. My approach is to import data in R and than use it in mx.model.FeedForward.create() .The code is here:
trsin_size <- 1000\ndata_train <- read.csv("./local_train-64x64-data.csv", header=TRUE)\ndata_train <- data.matrix(data_train)\ndata_train <- data_train[c(1:trsin_size) , ]; gc()\ndim(data_train)\n\n\nlabel_train_stytole <- read.csv("./local_train-stytole.csv", header=TRUE)\nlabel_train_stytole <- data.matrix(label_train_stytole)[c(1:trsin_size) , ]; gc()\ndim(label_train_stytole)\n\nmx.set.seed(0)\nstytole_model <- mx.model.FeedForward.create(\n X = data_train,\n y = label_train_stytole,\n ctx = mx.cpu(),\n symbol = network,\n num.round = 65,\n learning.rate = 0.001,\n wd = 0.00001,\n momentum = 0.9,\n eval.metric = mx.metric.CRPS,\n)\nBut this does not work. The mx.model.FeedForward.create() returns the following error message:
Auto detect layout of input matrix, use rowmajor..\nError: io.cc:54: Data and label shape in-consistent\nProbably due to the fact that in get.lenet() function frames <- mx.symbol.SliceChannel(source, num.outputs = 30) is used and this function requires appripiate shape of data. My question is how - if at all - is posible to shape train data in R in appropriate way to pass it to mxnet function mx.model.FeedForward.create() for implemented gen.len() network?
[quote=Franc Bračun;102868]
\n\n@Qiang: When do you plan to release a weekly update of windows package?
\n\n[/quote]
\n\nThe Windows expert in our team (he is from Microsoft) will update it soon.
", "rawMarkdown": "[quote=Franc Bračun;102868]\r\n\r\n@Qiang: When do you plan to release a weekly update of windows package?\r\n\r\n[/quote]\r\n\r\nThe Windows expert in our team (he is from Microsoft) will update it soon.", "votes": null }, { "id": "102896", "postDate": "12/27/2015 16:15:41", "content": "First of all, kudos to the entire dmlc crew - you guys are doing a truly amazing job.
\n\nI am trying to run the code using the pre-built version in R on Mac OSX and I was wondering if anybody could give me a hand:
\n\ndata_train <- mx.io.CSVIter(\n data.csv = "./train-64x64-data.csv", data.shape = c(64, 64, 30),\n label.csv = "./train-stytole.csv", label.shape = 600,\n batch.size = batch_size\n )\nand i get
\n\n[17:11:25] ./dmlc-core/include/dmlc/logging.h:208: [17:11:25] src/data.cc:30: unknown datatype csv\nError: [17:11:25] src/data.cc:30: unknown datatype csv\nThe weird thing is "?mx.io.CSVIter" does produce a valid help entry, so the function is clearly there ( package installation did have a zero exit status, after all) - and you would expect a function called CSVIter does recognize .csv as a format :-) I am honestly at a loss here, so any help would be very much appreciated.
", "rawMarkdown": "First of all, kudos to the entire dmlc crew - you guys are doing a truly amazing job. \r\n\r\nI am trying to run the code using the pre-built version in R on Mac OSX and I was wondering if anybody could give me a hand: \r\n\r\n data_train <- mx.io.CSVIter(\r\n data.csv = \"./train-64x64-data.csv\", data.shape = c(64, 64, 30),\r\n label.csv = \"./train-stytole.csv\", label.shape = 600,\r\n batch.size = batch_size\r\n )\r\n\r\nand i get \r\n\r\n [17:11:25] ./dmlc-core/include/dmlc/logging.h:208: [17:11:25] src/data.cc:30: unknown datatype csv\r\n Error: [17:11:25] src/data.cc:30: unknown datatype csv\r\n\r\n\r\nThe weird thing is \"?mx.io.CSVIter\" **does** produce a valid help entry, so the function is clearly there ( package installation did have a zero exit status, after all) - and you would expect a function called CSVIter does recognize .csv as a format :-) I am honestly at a loss here, so any help would be very much appreciated.", "votes": null }, { "id": "103044", "postDate": "12/28/2015 15:25:41", "content": "Our Windows expert just updated the Windows lib.
\n\nYou should have csviter in Windows now.
\n\n[quote=Franc Bračun;102868]
\n\n@Qiang: When do you plan to release a weekly update of windows package?
\n\n[/quote]
", "rawMarkdown": "Our Windows expert just updated the Windows lib.\r\n\r\nYou should have csviter in Windows now.\r\n\r\n[quote=Franc Bračun;102868]\r\n\r\n@Qiang: When do you plan to release a weekly update of windows package?\r\n\r\n[/quote]", "votes": null }, { "id": "103131", "postDate": "12/29/2015 07:15:27", "content": "@Qiang: Thanks. Kudos to the entire dmlc team.
", "rawMarkdown": "Qiang: Thanks. Kudos to the entire dmlc team.", "votes": null }, { "id": "103188", "postDate": "12/29/2015 23:11:36", "content": "Apologies for a potentially silly question, but where do the train-stytole.csv and train-diastole.csv files (used in lines 51 and 97, resp, of the R script) come from? They do not seem to be created by the Python preprocessing, and the size (600) is not in line with the train-labels size (> 5k).
", "rawMarkdown": "Apologies for a potentially silly question, but where do the train-stytole.csv and train-diastole.csv files (used in lines 51 and 97, resp, of the R script) come from? They do not seem to be created by the Python preprocessing, and the size (600) is not in line with the train-labels size (> 5k).", "votes": null }, { "id": "103203", "postDate": "12/30/2015 06:41:27", "content": "They are preprocessed in Train.py. I copied relevant parts from Train.py and pasted them into Python pre-processing file.
\n\nRelevant part of code is:
\n\n# In[3]:\n\ndef encode_label(label_data):\n """Run encoding to encode the label into the CDF target.\n """\n stytole = label_data[:, 1]\n diastole = label_data[:, 2]\n stytole_encode = np.array([\n (x < np.arange(600)) for x in stytole\n ], dtype=np.uint8)\n diastole_encode = np.array([\n (x < np.arange(600)) for x in diastole\n ], dtype=np.uint8)\n return stytole_encode, diastole_encode\n\ndef encode_csv(label_csv, stytole_csv, diastole_csv):\n stytole_encode, diastole_encode = encode_label(np.loadtxt(label_csv, delimiter=","))\n np.savetxt(stytole_csv, stytole_encode, delimiter=",", fmt="%g")\n np.savetxt(diastole_csv, diastole_encode, delimiter=",", fmt="%g")\n\n# Write encoded label into the target csv\n# We use CSV so that not all data need to sit into memory\n# You can also use inmemory numpy array if your machine is large enough\nencode_csv("./train-label.csv", "./train-stytole.csv", "./train-diastole.csv")\nHah! This one had not occurred to me. Thank you very much, Franc.
", "rawMarkdown": "Hah! This one had not occurred to me. Thank you very much, Franc.", "votes": null }, { "id": "103212", "postDate": "12/30/2015 07:47:21", "content": "I've had trouble installing mxnet in the first place for OSX Capitan. Full version of my question at StackOverflow: http://stackoverflow.com/questions/34525112/trouble-installing-mxnet-on-osx-capitan.
\n\nThe short story is that when I try to build mxnet on OSX Capitan I'm getting an error in the file /mxnet/dmlc-core/include/dmlc/base.h associated with sys/types.h which is not found. I've verified that I have this file in my c++ library and have tried manually recoding the path reference in base.h but only created more errors.
Any thoughts?
", "rawMarkdown": "I've had trouble installing mxnet in the first place for OSX Capitan. Full version of my question at StackOverflow: http://stackoverflow.com/questions/34525112/trouble-installing-mxnet-on-osx-capitan. \r\n\r\nThe short story is that when I try to build mxnet on OSX Capitan I'm getting an error in the file `/mxnet/dmlc-core/include/dmlc/base.h` associated with `sys/types.h` which is not found. I've verified that I have this file in my c++ library and have tried manually recoding the path reference in `base.h` but only created more errors.\r\n\r\nAny thoughts?", "votes": null }, { "id": "103236", "postDate": "12/30/2015 14:52:32", "content": "Hi, guys~~
\n\nJust a remainder. If you met any problems or questions, please feel free to open an issue on github.
\n\nhttps://github.com/dmlc/mxnet/issues
\n\nYou will get much much quicker and better response.
\n\nQiang Kou
", "rawMarkdown": "Hi, guys~~\r\n\r\nJust a remainder. If you met any problems or questions, please feel free to open an issue on github.\r\n\r\nhttps://github.com/dmlc/mxnet/issues\r\n\r\nYou will get much much quicker and better response.\r\n\r\nQiang Kou", "votes": null }, { "id": "103267", "postDate": "12/31/2015 01:25:21", "content": "The fix to the problem I was having was to update the ADD_CFLAGS variable in the make/osx.mk file. On my machine this looked like:
ADD_CFLAGS = -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -mmacosx-version-min=10.11
https://github.com/dmlc/mxnet/issues/1111
", "rawMarkdown": "The fix to the problem I was having was to update the ADD_CFLAGS variable in the `make/osx.mk file`. On my machine this looked like: \r\n\r\n`ADD_CFLAGS = -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -mmacosx-version-min=10.11`\r\n\r\nhttps://github.com/dmlc/mxnet/issues/1111", "votes": null }, { "id": "103331", "postDate": "01/01/2016 00:35:42", "content": "@Qiang
\n\nI'm having the same issue that @Konrad Banachewicz posted, where
\n\ndata_train <- mx.io.CSVIter(\n data.csv = "./train-64x64-data.csv", data.shape = c(64, 64, 30),\n label.csv = "./train-stytole.csv", label.shape = 600,\n batch.size = batch_size\n)\nProduces the error:
\n\n[16:29:28] ./dmlc-core/include/dmlc/logging.h:208: [16:29:28] src/data.cc:43: Unknown data type csv\nError: [16:29:28] src/data.cc:43: Unknown data type csv\nI've successfully compiled the mxnet and built the R package on OSX Capitan. I am using clang-omp as my CC export and clang-omp++ as my CXX export. Could this be causing my problem?
(Quick update: I just compiled mxnet today so it should be running the most current version. It wasn't clear to me whether the patch Qiang mentioned above for Windows was meant to apply here as well. installpackages(mxnet) method failed for me, so I built the package using option 2, from the command line, within the mxnet root folder)
Hi Qiang, I finally set everything up on my end and have run the mxnet scripts in both R and python. The only modifications I made to the code were to change mx.gpu to mx.cpu (I don't have an Invidia graphics card). I upped the number of training iterations from 65 to 100. Only as a point of information, the score I got with the code in R was ~0.08, while with python it was ~0.0385. I'll keep playing with it, but so far it looks like the Python code produces by far the best result.
", "rawMarkdown": "Hi Qiang, I finally set everything up on my end and have run the mxnet scripts in both R and python. The only modifications I made to the code were to change mx.gpu to mx.cpu (I don't have an Invidia graphics card). I upped the number of training iterations from 65 to 100. Only as a point of information, the score I got with the code in R was ~0.08, while with python it was ~0.0385. I'll keep playing with it, but so far it looks like the Python code produces by far the best result.", "votes": null }, { "id": "103831", "postDate": "01/06/2016 21:07:17", "content": "[quote=Aaron Polhamus;103829]
\n\nHi Qiang, I finally set everything up on my end and have run the mxnet scripts in both R and python. The only modifications I made to the code were to change mx.gpu to mx.cpu (I don't have an Invidia graphics card). I upped the number of training iterations from 65 to 100. Only as a point of information, the score I got with the code in R was ~0.08, while with python it was ~0.0385. I'll keep playing with it, but so far it looks like the Python code produces by far the best result.
\n\n[/quote]\nHi Aaron, how long does it take for your R code to run using cpu?
\n\nThanks!
", "rawMarkdown": "[quote=Aaron Polhamus;103829]\r\n\r\nHi Qiang, I finally set everything up on my end and have run the mxnet scripts in both R and python. The only modifications I made to the code were to change mx.gpu to mx.cpu (I don't have an Invidia graphics card). I upped the number of training iterations from 65 to 100. Only as a point of information, the score I got with the code in R was ~0.08, while with python it was ~0.0385. I'll keep playing with it, but so far it looks like the Python code produces by far the best result. \r\n\r\n[/quote]\r\nHi Aaron, how long does it take for your R code to run using cpu?\r\n\r\nThanks!", "votes": null }, { "id": "103834", "postDate": "01/06/2016 21:17:51", "content": "[quote=Aaron Polhamus;103829]
\n\nHi Qiang, I finally set everything up on my end and have run the mxnet scripts in both R and python. The only modifications I made to the code were to change mx.gpu to mx.cpu (I don't have an Invidia graphics card). I upped the number of training iterations from 65 to 100. Only as a point of information, the score I got with the code in R was ~0.08, while with python it was ~0.0385. I'll keep playing with it, but so far it looks like the Python code produces by far the best result.
\n\n[/quote]
\n\nIf the results are different, there might be something wrong.
\n\nLet me double check.
", "rawMarkdown": "[quote=Aaron Polhamus;103829]\r\n\r\nHi Qiang, I finally set everything up on my end and have run the mxnet scripts in both R and python. The only modifications I made to the code were to change mx.gpu to mx.cpu (I don't have an Invidia graphics card). I upped the number of training iterations from 65 to 100. Only as a point of information, the score I got with the code in R was ~0.08, while with python it was ~0.0385. I'll keep playing with it, but so far it looks like the Python code produces by far the best result. \r\n\r\n[/quote]\r\n\r\nIf the results are different, there might be something wrong.\r\n\r\nLet me double check.", "votes": null }, { "id": "103839", "postDate": "01/06/2016 22:01:14", "content": "I'm on an OSX quad core with 2.6 GHz Intel Core i7 CPUs. Total run time for 100 training iterations is ~6 - 8 hrs per model (so 12 - 16 for both systole and diastole).
", "rawMarkdown": "I'm on an OSX quad core with 2.6 GHz Intel Core i7 CPUs. Total run time for 100 training iterations is ~6 - 8 hrs per model (so 12 - 16 for both systole and diastole).", "votes": null }, { "id": "103881", "postDate": "01/07/2016 14:55:30", "content": "I have somewhat inline results with what is mentioned in the readme with the python script (~0.042 with no seed defined so I won't be able to reproduce) but not the R script (>0.20) which also seems to error on a cold start. Library built less than a week ago on yosemite. I will look into it and post updates to github issues.
", "rawMarkdown": "I have somewhat inline results with what is mentioned in the readme with the python script (~0.042 with no seed defined so I won't be able to reproduce) but not the R script (>0.20) which also seems to error on a cold start. Library built less than a week ago on yosemite. I will look into it and post updates to github issues.", "votes": null }, { "id": "103882", "postDate": "01/07/2016 14:59:36", "content": "FWIW: i had a similar experience with the R script (score > 0.2). I am curious if anybody replicated the Python result in R on a cpu.
", "rawMarkdown": "FWIW: i had a similar experience with the R script (score > 0.2). I am curious if anybody replicated the Python result in R on a cpu.", "votes": null }, { "id": "104186", "postDate": "01/10/2016 05:45:11", "content": "Hi,
\n\ni have run Train.R ,i got this error
Error: \n[10:24:36] include/dmlc/logging.h:208: [10:24:36] src/io/local_filesys.cc:61: LocalFileSystem.GetPathInfo ./train-stytole.csv Error:No such file or directory\nError: [10:24:36] src/io/local_filesys.cc:61: LocalFileSystem.GetPathInfo ./train-stytole.csv Error:No such file or directory
\n\nThis error is because of two files(train-stytole.csv , train-diastole.csv) are not generated by Preprocessing.py.\ncan anyone help me to generating these files\n 1) train-stytole.csv 2) train-diastole.csv ,or how to fix this error. thanks in advance.
Those files are generated by Train.py
\n\ndef encode_label(label_data):\n systole = label_data[:, 1]\n diastole = label_data[:, 2]\n systole_encode = np.array([\n (x < np.arange(600)) for x in systole\n ], dtype=np.uint8)\n diastole_encode = np.array([\n (x < np.arange(600)) for x in diastole\n ], dtype=np.uint8)\n return systole_encode, diastole_encode\n\ndef encode_csv(label_csv, systole_csv, diastole_csv):\n systole_encode, diastole_encode = encode_label(np.loadtxt(label_csv, delimiter=","))\n np.savetxt(systole_csv, systole_encode, delimiter=",", fmt="%g")\n np.savetxt(diastole_csv, diastole_encode, delimiter=",", fmt="%g")\n\nencode_csv("./train-label.csv", "./train-systole.csv", "./train-diastole.csv")\nI fixed the typo in "stytole" so you might want to revert that.
", "rawMarkdown": "Those files are generated by Train.py\r\n\r\n def encode_label(label_data):\r\n systole = label_data[:, 1]\r\n diastole = label_data[:, 2]\r\n systole_encode = np.array([\r\n (x < np.arange(600)) for x in systole\r\n ], dtype=np.uint8)\r\n diastole_encode = np.array([\r\n (x < np.arange(600)) for x in diastole\r\n ], dtype=np.uint8)\r\n return systole_encode, diastole_encode\r\n \r\n def encode_csv(label_csv, systole_csv, diastole_csv):\r\n systole_encode, diastole_encode = encode_label(np.loadtxt(label_csv, delimiter=\",\"))\r\n np.savetxt(systole_csv, systole_encode, delimiter=\",\", fmt=\"%g\")\r\n np.savetxt(diastole_csv, diastole_encode, delimiter=\",\", fmt=\"%g\")\r\n \r\n encode_csv(\"./train-label.csv\", \"./train-systole.csv\", \"./train-diastole.csv\")\r\n\r\nI fixed the typo in \"stytole\" so you might want to revert that.", "votes": null }, { "id": "104626", "postDate": "01/14/2016 18:14:11", "content": "would be nice if we can there is some pkgs in r that does the preprocessing pipelines as well.
", "rawMarkdown": "would be nice if we can there is some pkgs in r that does the preprocessing pipelines as well.", "votes": null }, { "id": "104627", "postDate": "01/14/2016 18:20:08", "content": "The preprocessing could be done in R, since we can read the data format
\n\nhttps://cran.r-project.org/web/packages/oro.dicom/index.html
", "rawMarkdown": "The preprocessing could be done in R, since we can read the data format\r\n\r\nhttps://cran.r-project.org/web/packages/oro.dicom/index.html", "votes": null }, { "id": "106671", "postDate": "02/03/2016 05:17:14", "content": "[quote=Qiang;104627]
\n\nThe preprocessing could be done in R, since we can read the data format
\n\nhttps://cran.r-project.org/web/packages/oro.dicom/index.html
\n\n[/quote]
\n\nThis seems to be nice, any r hackers who want to take a stab?
", "rawMarkdown": "[quote=Qiang;104627]\r\n\r\nThe preprocessing could be done in R, since we can read the data format\r\n\r\nhttps://cran.r-project.org/web/packages/oro.dicom/index.html\r\n\r\n[/quote]\r\n\r\nThis seems to be nice, any r hackers who want to take a stab?", "votes": null }, { "id": "120625", "postDate": "05/19/2016 15:50:51", "content": "I have been able to get this to work a few weeks ago, but now I'm getting an error running:
\n\nnormed <- preproc.image(im, mean.img)
\n\nError: Expecting a four-dimensional array
\n\nI had to previously convert the parrots.png file to parrots.bmp to get the process to work,\nbut now I get the same error when using the png file.
\n\nThank you.
", "rawMarkdown": "I have been able to get this to work a few weeks ago, but now I'm getting an error running:\r\n\r\nnormed <- preproc.image(im, mean.img)\r\n\r\n Error: Expecting a four-dimensional array\r\n\r\nI had to previously convert the parrots.png file to parrots.bmp to get the process to work,\r\nbut now I get the same error when using the png file.\r\n\r\nThank you.", "votes": null }, { "id": "120642", "postDate": "05/19/2016 18:27:18", "content": "[quote=tjclifford;120625]
\n\nI have been able to get this to work a few weeks ago, but now I'm getting an error running:
\n\nnormed <- preproc.image(im, mean.img)
\n\nError: Expecting a four-dimensional array
\n\nI had to previously convert the parrots.png file to parrots.bmp to get the process to work,\nbut now I get the same error when using the png file.
\n\nThank you.
\n\n[/quote]
\n\nYou met this error because imager changed their APIs.
We have also updated the code to follow their changes, please see the latest documents: http://mxnet.dmlc.ml/en/latest/packages/r/classifyRealImageWithPretrainedModel.html
", "rawMarkdown": "[quote=tjclifford;120625]\r\n\r\nI have been able to get this to work a few weeks ago, but now I'm getting an error running:\r\n\r\nnormed <- preproc.image(im, mean.img)\r\n\r\n Error: Expecting a four-dimensional array\r\n\r\nI had to previously convert the parrots.png file to parrots.bmp to get the process to work,\r\nbut now I get the same error when using the png file.\r\n\r\nThank you.\r\n\r\n[/quote]\r\n\r\nYou met this error because `imager` changed their APIs.\r\n\r\nWe have also updated the code to follow their changes, please see the latest documents: http://mxnet.dmlc.ml/en/latest/packages/r/classifyRealImageWithPretrainedModel.html", "votes": null }, { "id": "125615", "postDate": "06/30/2016 21:16:59", "content": "Hey,
\n\ncan't find any help so i'm posting this here, hoping some of you'll be able to enlight my day!\nhttps://github.com/dmlc/mxnet/issues/2535
\n\nJacques,
", "rawMarkdown": "Hey,\r\n\r\ncan't find any help so i'm posting this here, hoping some of you'll be able to enlight my day!\r\nhttps://github.com/dmlc/mxnet/issues/2535\r\n\r\nJacques,", "votes": null }, { "id": "132144", "postDate": "08/22/2016 22:23:08", "content": "\n\n\nYou met this error because imager changed their APIs.
\n \nWe have also updated the code to follow their changes, please see the\n latest documents:\n http://mxnet.dmlc.ml/en/latest/packages/r/classifyRealImageWithPretrainedModel.html
\n
This link no longer works and I really hope to try this out. Is there a new link with the fix to the imager API? Thanks!
", "rawMarkdown": "> You met this error because imager changed their APIs.\r\n> \r\n> We have also updated the code to follow their changes, please see the\r\n> latest documents:\r\n> http://mxnet.dmlc.ml/en/latest/packages/r/classifyRealImageWithPretrainedModel.html\r\n\r\n\r\nThis link no longer works and I really hope to try this out. Is there a new link with the fix to the imager API? Thanks!", "votes": null }, { "id": "161653", "postDate": "02/15/2017 02:51:37", "content": "Good news for R users!
", "rawMarkdown": "Good news for R users!", "votes": null } ], "comments": [ { "id": 102745, "author_name": "francbracun", "author_url": "", "post_date": "12/25/2015 09:20:20", "content": "Thumbs up for MxNet team! Great job.
\n\nOne probem appeared to me. When using the code I get the following errors:
\n\n> source <- (source-128) / 128\nError in source - 128: non-numeric argument to a binary operator\nand
\n\n> network <- get.lenet()\nError in frames[[i + 1]] : wrong arguments for subsetting an environment\nHi guys! I am trying to install the mxnet package, and it fails with the following:
\n\n\n\n\ndrat:::addRepo("dmlc")\n install.packages("mxnet")\n Warning in install.packages :\n cannot open URL 'http://dmlc.github.io/drat/src/contrib/PACKAGES.gz': HTTP status was '404 Not Found'\n Warning in install.packages :\n cannot open URL 'http://dmlc.github.io/drat/src/contrib/PACKAGES': HTTP status was '404 Not Found'\n Warning in install.packages :\n unable to access index for repository http://dmlc.github.io/drat/src/contrib:\n cannot open URL 'http://dmlc.github.io/drat/src/contrib/PACKAGES'
\n
I have tried to also access the URLs above, and they don't work. Any thoughts?
\n\nMany thanks!
", "votes": null, "replies": [] }, { "id": 102751, "author_name": "francbracun", "author_url": "", "post_date": "12/25/2015 13:58:32", "content": "I have done the following and it works for me:
\n\ndrat package from CRANE (I have done this in RStudio: Tools / Install Packages)drat:::addRepo("dmlc") in R Shellinstall.packages("mxnet") in R shellI have done the following and it works for me:
\n\ndrat package from CRANE (I have done this in RStudio: Tools / Install Packages)\nWhen I used install.packages("drat", repos="https://cran.rstudio.com") in R Shell, I've got simmilar errordrat:::addRepo("dmlc") in R Shellinstall.packages("mxnet") in R shellThanks a lot, Franc! It works in this way. I am also getting the errors that you described above.\nFor the 1st one, I did something like this and got past it (not sure if it's correct though):\n source <- mx.symbol.Variable("data")\n source2 <- mx.symbol.Variable("128")\n source <- (source-source2) / source2
\n\nHowever, I did not find any way around the 2nd one and moreover this one:\n Error: could not find function "mx.io.CSVIter". The python MxNet version also has this issue.\nThat is because the pre-built Windows package is behind the master from the repo.\nI don't have cuDNN yet (waiting for approval of NVidia user) so I guess I will wait for the next pre-built package for both Python & R. Hope the MxNet team will upload it soon.
", "votes": null, "replies": [] }, { "id": 102757, "author_name": "thirdwing", "author_url": "", "post_date": "12/25/2015 16:35:11", "content": "Hi, guys,
\n\nSorry for being a little late. BlackCore is correct, the prebuilt packages are updated weekly, so it might be behind the github master.
\n\nI have just updated the Mac and Windows package. You should be able to install it.
\n\nNote: all prebuilt packages are CPU-only. If you want to use your GPU or you are a Linux hacker, you should compile it by yourself.
", "votes": null, "replies": [] }, { "id": 102759, "author_name": "francbracun", "author_url": "", "post_date": "12/25/2015 16:51:29", "content": "Thanks Qiang. Now the above errors have gone. :)
\n\nThere is a new error that I have faced:
\n\n> net <- mx.symbol.BatchNorm(net, fix.gamma = TRUE)\nError: Cannot find argument 'fix_gamma', Possible Arguments:\n----------------\neps : float, optional, default=1e-010\n Epsilon to prevent div 0\nmomentum : float, optional, default=0.9\n Momentum for moving average\nHowever, when I use:
\n\n net <- mx.symbol.BatchNorm(data=net, eps=0.001, momentum=0.9 )\nthere is no error returned by this function. I suspect that the TRUE is not appropriate parameter for fix.gamma.
The next error I have faced and I do not know, what is the possible reason:
\n\n> data_train <- mx.io.CSVIter(\n+ data.csv = "./train-64x64-data.csv", data.shape = c(64, 64, 30),\n+ label.csv = "./train-stytole.csv", label.shape = 600,\n+ batch.size = batch_size\n+ )\nError in mx.io.CSVIter(data.csv = "./train-64x64-data.csv", data.shape = c(64, : \n could not find function "mx.varg.io.CSVIter"\nI think both of errors you met are due to the Windows lib is behind github master. Similar situations in python, since both of R and python are using the same Windows lib file.
\n\nPlease wait for some time.
", "votes": null, "replies": [] }, { "id": 102783, "author_name": "francbracun", "author_url": "", "post_date": "12/25/2015 22:50:41", "content": "Thanks.
", "votes": null, "replies": [] }, { "id": 102803, "author_name": "phunter", "author_url": "", "post_date": "12/26/2015 06:45:57", "content": "mxnet rocks with R!
", "votes": null, "replies": [] }, { "id": 102868, "author_name": "francbracun", "author_url": "", "post_date": "12/27/2015 06:55:24", "content": "@Qiang: When do you plan to release a weekly update of windows package?
", "votes": null, "replies": [] }, { "id": 102869, "author_name": "francbracun", "author_url": "", "post_date": "12/27/2015 06:55:41", "content": "Did any one try to implement in-memory solution. My approach is to import data in R and than use it in mx.model.FeedForward.create() .The code is here:
trsin_size <- 1000\ndata_train <- read.csv("./local_train-64x64-data.csv", header=TRUE)\ndata_train <- data.matrix(data_train)\ndata_train <- data_train[c(1:trsin_size) , ]; gc()\ndim(data_train)\n\n\nlabel_train_stytole <- read.csv("./local_train-stytole.csv", header=TRUE)\nlabel_train_stytole <- data.matrix(label_train_stytole)[c(1:trsin_size) , ]; gc()\ndim(label_train_stytole)\n\nmx.set.seed(0)\nstytole_model <- mx.model.FeedForward.create(\n X = data_train,\n y = label_train_stytole,\n ctx = mx.cpu(),\n symbol = network,\n num.round = 65,\n learning.rate = 0.001,\n wd = 0.00001,\n momentum = 0.9,\n eval.metric = mx.metric.CRPS,\n)\nBut this does not work. The mx.model.FeedForward.create() returns the following error message:
Auto detect layout of input matrix, use rowmajor..\nError: io.cc:54: Data and label shape in-consistent\nProbably due to the fact that in get.lenet() function frames <- mx.symbol.SliceChannel(source, num.outputs = 30) is used and this function requires appripiate shape of data. My question is how - if at all - is posible to shape train data in R in appropriate way to pass it to mxnet function mx.model.FeedForward.create() for implemented gen.len() network?
[quote=Franc Bračun;102868]
\n\n@Qiang: When do you plan to release a weekly update of windows package?
\n\n[/quote]
\n\nThe Windows expert in our team (he is from Microsoft) will update it soon.
", "votes": null, "replies": [] }, { "id": 102896, "author_name": "konradb", "author_url": "", "post_date": "12/27/2015 16:15:41", "content": "First of all, kudos to the entire dmlc crew - you guys are doing a truly amazing job.
\n\nI am trying to run the code using the pre-built version in R on Mac OSX and I was wondering if anybody could give me a hand:
\n\ndata_train <- mx.io.CSVIter(\n data.csv = "./train-64x64-data.csv", data.shape = c(64, 64, 30),\n label.csv = "./train-stytole.csv", label.shape = 600,\n batch.size = batch_size\n )\nand i get
\n\n[17:11:25] ./dmlc-core/include/dmlc/logging.h:208: [17:11:25] src/data.cc:30: unknown datatype csv\nError: [17:11:25] src/data.cc:30: unknown datatype csv\nThe weird thing is "?mx.io.CSVIter" does produce a valid help entry, so the function is clearly there ( package installation did have a zero exit status, after all) - and you would expect a function called CSVIter does recognize .csv as a format :-) I am honestly at a loss here, so any help would be very much appreciated.
", "votes": null, "replies": [] }, { "id": 103044, "author_name": "thirdwing", "author_url": "", "post_date": "12/28/2015 15:25:41", "content": "Our Windows expert just updated the Windows lib.
\n\nYou should have csviter in Windows now.
\n\n[quote=Franc Bračun;102868]
\n\n@Qiang: When do you plan to release a weekly update of windows package?
\n\n[/quote]
", "votes": null, "replies": [] }, { "id": 103131, "author_name": "francbracun", "author_url": "", "post_date": "12/29/2015 07:15:27", "content": "@Qiang: Thanks. Kudos to the entire dmlc team.
", "votes": null, "replies": [] }, { "id": 103188, "author_name": "konradb", "author_url": "", "post_date": "12/29/2015 23:11:36", "content": "Apologies for a potentially silly question, but where do the train-stytole.csv and train-diastole.csv files (used in lines 51 and 97, resp, of the R script) come from? They do not seem to be created by the Python preprocessing, and the size (600) is not in line with the train-labels size (> 5k).
", "votes": null, "replies": [] }, { "id": 103203, "author_name": "francbracun", "author_url": "", "post_date": "12/30/2015 06:41:27", "content": "They are preprocessed in Train.py. I copied relevant parts from Train.py and pasted them into Python pre-processing file.
\n\nRelevant part of code is:
\n\n# In[3]:\n\ndef encode_label(label_data):\n """Run encoding to encode the label into the CDF target.\n """\n stytole = label_data[:, 1]\n diastole = label_data[:, 2]\n stytole_encode = np.array([\n (x < np.arange(600)) for x in stytole\n ], dtype=np.uint8)\n diastole_encode = np.array([\n (x < np.arange(600)) for x in diastole\n ], dtype=np.uint8)\n return stytole_encode, diastole_encode\n\ndef encode_csv(label_csv, stytole_csv, diastole_csv):\n stytole_encode, diastole_encode = encode_label(np.loadtxt(label_csv, delimiter=","))\n np.savetxt(stytole_csv, stytole_encode, delimiter=",", fmt="%g")\n np.savetxt(diastole_csv, diastole_encode, delimiter=",", fmt="%g")\n\n# Write encoded label into the target csv\n# We use CSV so that not all data need to sit into memory\n# You can also use inmemory numpy array if your machine is large enough\nencode_csv("./train-label.csv", "./train-stytole.csv", "./train-diastole.csv")\nHah! This one had not occurred to me. Thank you very much, Franc.
", "votes": null, "replies": [] }, { "id": 103212, "author_name": "aaronpolhamus", "author_url": "", "post_date": "12/30/2015 07:47:21", "content": "I've had trouble installing mxnet in the first place for OSX Capitan. Full version of my question at StackOverflow: http://stackoverflow.com/questions/34525112/trouble-installing-mxnet-on-osx-capitan.
\n\nThe short story is that when I try to build mxnet on OSX Capitan I'm getting an error in the file /mxnet/dmlc-core/include/dmlc/base.h associated with sys/types.h which is not found. I've verified that I have this file in my c++ library and have tried manually recoding the path reference in base.h but only created more errors.
Any thoughts?
", "votes": null, "replies": [] }, { "id": 103236, "author_name": "thirdwing", "author_url": "", "post_date": "12/30/2015 14:52:32", "content": "Hi, guys~~
\n\nJust a remainder. If you met any problems or questions, please feel free to open an issue on github.
\n\nhttps://github.com/dmlc/mxnet/issues
\n\nYou will get much much quicker and better response.
\n\nQiang Kou
", "votes": null, "replies": [] }, { "id": 103267, "author_name": "aaronpolhamus", "author_url": "", "post_date": "12/31/2015 01:25:21", "content": "The fix to the problem I was having was to update the ADD_CFLAGS variable in the make/osx.mk file. On my machine this looked like:
ADD_CFLAGS = -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -mmacosx-version-min=10.11
https://github.com/dmlc/mxnet/issues/1111
", "votes": null, "replies": [] }, { "id": 103331, "author_name": "aaronpolhamus", "author_url": "", "post_date": "01/01/2016 00:35:42", "content": "@Qiang
\n\nI'm having the same issue that @Konrad Banachewicz posted, where
\n\ndata_train <- mx.io.CSVIter(\n data.csv = "./train-64x64-data.csv", data.shape = c(64, 64, 30),\n label.csv = "./train-stytole.csv", label.shape = 600,\n batch.size = batch_size\n)\nProduces the error:
\n\n[16:29:28] ./dmlc-core/include/dmlc/logging.h:208: [16:29:28] src/data.cc:43: Unknown data type csv\nError: [16:29:28] src/data.cc:43: Unknown data type csv\nI've successfully compiled the mxnet and built the R package on OSX Capitan. I am using clang-omp as my CC export and clang-omp++ as my CXX export. Could this be causing my problem?
(Quick update: I just compiled mxnet today so it should be running the most current version. It wasn't clear to me whether the patch Qiang mentioned above for Windows was meant to apply here as well. installpackages(mxnet) method failed for me, so I built the package using option 2, from the command line, within the mxnet root folder)
Hi Qiang, I finally set everything up on my end and have run the mxnet scripts in both R and python. The only modifications I made to the code were to change mx.gpu to mx.cpu (I don't have an Invidia graphics card). I upped the number of training iterations from 65 to 100. Only as a point of information, the score I got with the code in R was ~0.08, while with python it was ~0.0385. I'll keep playing with it, but so far it looks like the Python code produces by far the best result.
", "votes": null, "replies": [] }, { "id": 103831, "author_name": "russwill", "author_url": "", "post_date": "01/06/2016 21:07:17", "content": "[quote=Aaron Polhamus;103829]
\n\nHi Qiang, I finally set everything up on my end and have run the mxnet scripts in both R and python. The only modifications I made to the code were to change mx.gpu to mx.cpu (I don't have an Invidia graphics card). I upped the number of training iterations from 65 to 100. Only as a point of information, the score I got with the code in R was ~0.08, while with python it was ~0.0385. I'll keep playing with it, but so far it looks like the Python code produces by far the best result.
\n\n[/quote]\nHi Aaron, how long does it take for your R code to run using cpu?
\n\nThanks!
", "votes": null, "replies": [] }, { "id": 103834, "author_name": "thirdwing", "author_url": "", "post_date": "01/06/2016 21:17:51", "content": "[quote=Aaron Polhamus;103829]
\n\nHi Qiang, I finally set everything up on my end and have run the mxnet scripts in both R and python. The only modifications I made to the code were to change mx.gpu to mx.cpu (I don't have an Invidia graphics card). I upped the number of training iterations from 65 to 100. Only as a point of information, the score I got with the code in R was ~0.08, while with python it was ~0.0385. I'll keep playing with it, but so far it looks like the Python code produces by far the best result.
\n\n[/quote]
\n\nIf the results are different, there might be something wrong.
\n\nLet me double check.
", "votes": null, "replies": [] }, { "id": 103839, "author_name": "aaronpolhamus", "author_url": "", "post_date": "01/06/2016 22:01:14", "content": "I'm on an OSX quad core with 2.6 GHz Intel Core i7 CPUs. Total run time for 100 training iterations is ~6 - 8 hrs per model (so 12 - 16 for both systole and diastole).
", "votes": null, "replies": [] }, { "id": 103881, "author_name": "francoislelay", "author_url": "", "post_date": "01/07/2016 14:55:30", "content": "I have somewhat inline results with what is mentioned in the readme with the python script (~0.042 with no seed defined so I won't be able to reproduce) but not the R script (>0.20) which also seems to error on a cold start. Library built less than a week ago on yosemite. I will look into it and post updates to github issues.
", "votes": null, "replies": [] }, { "id": 103882, "author_name": "konradb", "author_url": "", "post_date": "01/07/2016 14:59:36", "content": "FWIW: i had a similar experience with the R script (score > 0.2). I am curious if anybody replicated the Python result in R on a cpu.
", "votes": null, "replies": [] }, { "id": 104186, "author_name": "niranjanmudhiraj", "author_url": "", "post_date": "01/10/2016 05:45:11", "content": "Hi,
\n\ni have run Train.R ,i got this error
Error: \n[10:24:36] include/dmlc/logging.h:208: [10:24:36] src/io/local_filesys.cc:61: LocalFileSystem.GetPathInfo ./train-stytole.csv Error:No such file or directory\nError: [10:24:36] src/io/local_filesys.cc:61: LocalFileSystem.GetPathInfo ./train-stytole.csv Error:No such file or directory
\n\nThis error is because of two files(train-stytole.csv , train-diastole.csv) are not generated by Preprocessing.py.\ncan anyone help me to generating these files\n 1) train-stytole.csv 2) train-diastole.csv ,or how to fix this error. thanks in advance.
Those files are generated by Train.py
\n\ndef encode_label(label_data):\n systole = label_data[:, 1]\n diastole = label_data[:, 2]\n systole_encode = np.array([\n (x < np.arange(600)) for x in systole\n ], dtype=np.uint8)\n diastole_encode = np.array([\n (x < np.arange(600)) for x in diastole\n ], dtype=np.uint8)\n return systole_encode, diastole_encode\n\ndef encode_csv(label_csv, systole_csv, diastole_csv):\n systole_encode, diastole_encode = encode_label(np.loadtxt(label_csv, delimiter=","))\n np.savetxt(systole_csv, systole_encode, delimiter=",", fmt="%g")\n np.savetxt(diastole_csv, diastole_encode, delimiter=",", fmt="%g")\n\nencode_csv("./train-label.csv", "./train-systole.csv", "./train-diastole.csv")\nI fixed the typo in "stytole" so you might want to revert that.
", "votes": null, "replies": [] }, { "id": 104626, "author_name": "tqchen", "author_url": "", "post_date": "01/14/2016 18:14:11", "content": "would be nice if we can there is some pkgs in r that does the preprocessing pipelines as well.
", "votes": null, "replies": [] }, { "id": 104627, "author_name": "thirdwing", "author_url": "", "post_date": "01/14/2016 18:20:08", "content": "The preprocessing could be done in R, since we can read the data format
\n\nhttps://cran.r-project.org/web/packages/oro.dicom/index.html
", "votes": null, "replies": [] }, { "id": 106671, "author_name": "tqchen", "author_url": "", "post_date": "02/03/2016 05:17:14", "content": "[quote=Qiang;104627]
\n\nThe preprocessing could be done in R, since we can read the data format
\n\nhttps://cran.r-project.org/web/packages/oro.dicom/index.html
\n\n[/quote]
\n\nThis seems to be nice, any r hackers who want to take a stab?
", "votes": null, "replies": [] }, { "id": 120625, "author_name": "tjclifford", "author_url": "", "post_date": "05/19/2016 15:50:51", "content": "I have been able to get this to work a few weeks ago, but now I'm getting an error running:
\n\nnormed <- preproc.image(im, mean.img)
\n\nError: Expecting a four-dimensional array
\n\nI had to previously convert the parrots.png file to parrots.bmp to get the process to work,\nbut now I get the same error when using the png file.
\n\nThank you.
", "votes": null, "replies": [] }, { "id": 120642, "author_name": "thirdwing", "author_url": "", "post_date": "05/19/2016 18:27:18", "content": "[quote=tjclifford;120625]
\n\nI have been able to get this to work a few weeks ago, but now I'm getting an error running:
\n\nnormed <- preproc.image(im, mean.img)
\n\nError: Expecting a four-dimensional array
\n\nI had to previously convert the parrots.png file to parrots.bmp to get the process to work,\nbut now I get the same error when using the png file.
\n\nThank you.
\n\n[/quote]
\n\nYou met this error because imager changed their APIs.
We have also updated the code to follow their changes, please see the latest documents: http://mxnet.dmlc.ml/en/latest/packages/r/classifyRealImageWithPretrainedModel.html
", "votes": null, "replies": [] }, { "id": 125615, "author_name": "jacquespeeters", "author_url": "", "post_date": "06/30/2016 21:16:59", "content": "Hey,
\n\ncan't find any help so i'm posting this here, hoping some of you'll be able to enlight my day!\nhttps://github.com/dmlc/mxnet/issues/2535
\n\nJacques,
", "votes": null, "replies": [] }, { "id": 132144, "author_name": "slimthealien", "author_url": "", "post_date": "08/22/2016 22:23:08", "content": "\n\n\nYou met this error because imager changed their APIs.
\n \nWe have also updated the code to follow their changes, please see the\n latest documents:\n http://mxnet.dmlc.ml/en/latest/packages/r/classifyRealImageWithPretrainedModel.html
\n
This link no longer works and I really hope to try this out. Is there a new link with the fix to the imager API? Thanks!
", "votes": null, "replies": [] }, { "id": 161653, "author_name": "franksdnu", "author_url": "", "post_date": "02/15/2017 02:51:37", "content": "Good news for R users!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "102737": "Hi, all, \r\n\r\nWe are providing a deep learning model in R using the `mxnet` package. \r\n\r\nhttps://github.com/dmlc/mxnet/blob/master/example/kaggle-ndsb2/Train.R\r\n\r\nIt is the same network structure and parameters with the model from Bing Xu. And of course they have the same lb score.\r\n\r\nAs a R user for years, I personally feel a little unhappy that we don't have good deep learning tools in R. Even the deep learning tutorials on Kaggle are all using python.\r\n\r\nNow we have `mxnet`! \r\n\r\nTo install the `mxnet` R package, please follow the instructions from:\r\n\r\nhttp://mxnet.readthedocs.org/en/latest/build.html#r-package-installation\r\n\r\nWe also provide several tutorials:\r\n\r\nhttp://mxnet.readthedocs.org/en/latest/R-package/index.html#tutorials\r\n\r\nRight now the model is using pre-processed csv files by python code. I will add the preprocessing R code later.", "102745": "Thumbs up for MxNet team! Great job.\r\n\r\nOne probem appeared to me. When using the code I get the following errors:\r\n\r\n > source <- (source-128) / 128\r\n Error in source - 128: non-numeric argument to a binary operator\r\n\r\nand\r\n\r\n > network <- get.lenet()\r\n Error in frames[[i + 1]] : wrong arguments for subsetting an environment", "102746": "Hi guys! I am trying to install the mxnet package, and it fails with the following:\r\n\r\n> drat:::addRepo(\"dmlc\")\r\n> install.packages(\"mxnet\")\r\nWarning in install.packages :\r\n cannot open URL 'http://dmlc.github.io/drat/src/contrib/PACKAGES.gz': HTTP status was '404 Not Found'\r\nWarning in install.packages :\r\n cannot open URL 'http://dmlc.github.io/drat/src/contrib/PACKAGES': HTTP status was '404 Not Found'\r\nWarning in install.packages :\r\n unable to access index for repository http://dmlc.github.io/drat/src/contrib:\r\n cannot open URL 'http://dmlc.github.io/drat/src/contrib/PACKAGES'\r\n\r\nI have tried to also access the URLs above, and they don't work. Any thoughts?\r\n\r\nMany thanks!", "102751": "I have done the following and it works for me:\r\n\r\n1. install `drat` package **from CRANE** (I have done this in RStudio: Tools / Install Packages)\r\n2. `drat:::addRepo(\"dmlc\")` in R Shell\r\n3. `install.packages(\"mxnet\")` in R shell", "102752": "I have done the following and it works for me:\r\n\r\n1. install `drat` package **from CRANE** (I have done this in RStudio: Tools / Install Packages)\r\n When I used `install.packages(\"drat\", repos=\"https://cran.rstudio.com\")` **in R Shell**, I've got simmilar error\r\n2. `drat:::addRepo(\"dmlc\")` in R Shell\r\n3. `install.packages(\"mxnet\")` in R shell", "102754": "Thanks a lot, Franc! It works in this way. I am also getting the errors that you described above.\r\nFor the 1st one, I did something like this and got past it (not sure if it's correct though):\r\n source <- mx.symbol.Variable(\"data\")\r\n source2 <- mx.symbol.Variable(\"128\")\r\n source <- (source-source2) / source2\r\n\r\nHowever, I did not find any way around the 2nd one and moreover this one:\r\n Error: could not find function \"mx.io.CSVIter\". The python MxNet version also has this issue.\r\nThat is because the pre-built Windows package is behind the master from the repo.\r\nI don't have cuDNN yet (waiting for approval of NVidia user) so I guess I will wait for the next pre-built package for both Python & R. Hope the MxNet team will upload it soon.", "102757": "Hi, guys,\r\n\r\nSorry for being a little late. BlackCore is correct, the prebuilt packages are updated weekly, so it might be behind the github master.\r\n\r\nI have just updated the Mac and Windows package. You should be able to install it.\r\n\r\nNote: all prebuilt packages are CPU-only. If you want to use your GPU or you are a Linux hacker, you should compile it by yourself.", "102759": "Thanks Qiang. Now the above errors have gone. :) \r\n\r\nThere is a new error that I have faced:\r\n\r\n > net <- mx.symbol.BatchNorm(net, fix.gamma = TRUE)\r\n Error: Cannot find argument 'fix_gamma', Possible Arguments:\r\n ----------------\r\n eps : float, optional, default=1e-010\r\n Epsilon to prevent div 0\r\n momentum : float, optional, default=0.9\r\n Momentum for moving average\r\n\r\nHowever, when I use:\r\n\r\n net <- mx.symbol.BatchNorm(data=net, eps=0.001, momentum=0.9 )\r\n\r\nthere is no error returned by this function. I suspect that the `TRUE` is not appropriate parameter for `fix.gamma`.", "102760": "The next error I have faced and I do not know, what is the possible reason:\r\n\r\n > data_train <- mx.io.CSVIter(\r\n + data.csv = \"./train-64x64-data.csv\", data.shape = c(64, 64, 30),\r\n + label.csv = \"./train-stytole.csv\", label.shape = 600,\r\n + batch.size = batch_size\r\n + )\r\n Error in mx.io.CSVIter(data.csv = \"./train-64x64-data.csv\", data.shape = c(64, : \r\n could not find function \"mx.varg.io.CSVIter\"", "102769": "I think both of errors you met are due to the Windows lib is behind github master. Similar situations in python, since both of R and python are using the same Windows lib file.\r\n\r\nPlease wait for some time.", "102783": "Thanks.", "102803": "mxnet rocks with R!", "102868": "Qiang: When do you plan to release a weekly update of windows package?", "102869": "Did any one try to implement in-memory solution. My approach is to import data in R and than use it in `mx.model.FeedForward.create()` .The code is here:\r\n\r\n trsin_size <- 1000\r\n data_train <- read.csv(\"./local_train-64x64-data.csv\", header=TRUE)\r\n data_train <- data.matrix(data_train)\r\n data_train <- data_train[c(1:trsin_size) , ]; gc()\r\n dim(data_train)\r\n \r\n \r\n label_train_stytole <- read.csv(\"./local_train-stytole.csv\", header=TRUE)\r\n label_train_stytole <- data.matrix(label_train_stytole)[c(1:trsin_size) , ]; gc()\r\n dim(label_train_stytole)\r\n\r\n mx.set.seed(0)\r\n stytole_model <- mx.model.FeedForward.create(\r\n X = data_train,\r\n y = label_train_stytole,\r\n ctx = mx.cpu(),\r\n symbol = network,\r\n num.round = 65,\r\n learning.rate = 0.001,\r\n wd = 0.00001,\r\n momentum = 0.9,\r\n eval.metric = mx.metric.CRPS,\r\n )\r\n\r\n\r\nBut this does not work. The `mx.model.FeedForward.create()` returns the following error message:\r\n\r\n Auto detect layout of input matrix, use rowmajor..\r\n Error: io.cc:54: Data and label shape in-consistent\r\n\r\nProbably due to the fact that in `get.lenet()` function `frames <- mx.symbol.SliceChannel(source, num.outputs = 30)` is used and this function requires appripiate shape of data. My question is how - if at all - is posible to shape train data in R in appropriate way to pass it to `mxnet` function `mx.model.FeedForward.create()` for implemented `gen.len()` network?", "102894": "[quote=Franc Bračun;102868]\r\n\r\n@Qiang: When do you plan to release a weekly update of windows package?\r\n\r\n[/quote]\r\n\r\nThe Windows expert in our team (he is from Microsoft) will update it soon.", "102896": "First of all, kudos to the entire dmlc crew - you guys are doing a truly amazing job. \r\n\r\nI am trying to run the code using the pre-built version in R on Mac OSX and I was wondering if anybody could give me a hand: \r\n\r\n data_train <- mx.io.CSVIter(\r\n data.csv = \"./train-64x64-data.csv\", data.shape = c(64, 64, 30),\r\n label.csv = \"./train-stytole.csv\", label.shape = 600,\r\n batch.size = batch_size\r\n )\r\n\r\nand i get \r\n\r\n [17:11:25] ./dmlc-core/include/dmlc/logging.h:208: [17:11:25] src/data.cc:30: unknown datatype csv\r\n Error: [17:11:25] src/data.cc:30: unknown datatype csv\r\n\r\n\r\nThe weird thing is \"?mx.io.CSVIter\" **does** produce a valid help entry, so the function is clearly there ( package installation did have a zero exit status, after all) - and you would expect a function called CSVIter does recognize .csv as a format :-) I am honestly at a loss here, so any help would be very much appreciated.", "103044": "Our Windows expert just updated the Windows lib.\r\n\r\nYou should have csviter in Windows now.\r\n\r\n[quote=Franc Bračun;102868]\r\n\r\n@Qiang: When do you plan to release a weekly update of windows package?\r\n\r\n[/quote]", "103131": "Qiang: Thanks. Kudos to the entire dmlc team.", "103188": "Apologies for a potentially silly question, but where do the train-stytole.csv and train-diastole.csv files (used in lines 51 and 97, resp, of the R script) come from? They do not seem to be created by the Python preprocessing, and the size (600) is not in line with the train-labels size (> 5k).", "103203": "They are preprocessed in Train.py. I copied relevant parts from Train.py and pasted them into Python pre-processing file.\r\n\r\nRelevant part of code is:\r\n\r\n # In[3]:\r\n \r\n def encode_label(label_data):\r\n \"\"\"Run encoding to encode the label into the CDF target.\r\n \"\"\"\r\n stytole = label_data[:, 1]\r\n diastole = label_data[:, 2]\r\n stytole_encode = np.array([\r\n (x < np.arange(600)) for x in stytole\r\n ], dtype=np.uint8)\r\n diastole_encode = np.array([\r\n (x < np.arange(600)) for x in diastole\r\n ], dtype=np.uint8)\r\n return stytole_encode, diastole_encode\r\n \r\n def encode_csv(label_csv, stytole_csv, diastole_csv):\r\n stytole_encode, diastole_encode = encode_label(np.loadtxt(label_csv, delimiter=\",\"))\r\n np.savetxt(stytole_csv, stytole_encode, delimiter=\",\", fmt=\"%g\")\r\n np.savetxt(diastole_csv, diastole_encode, delimiter=\",\", fmt=\"%g\")\r\n \r\n # Write encoded label into the target csv\r\n # We use CSV so that not all data need to sit into memory\r\n # You can also use inmemory numpy array if your machine is large enough\r\n encode_csv(\"./train-label.csv\", \"./train-stytole.csv\", \"./train-diastole.csv\")", "103209": "Hah! This one had not occurred to me. Thank you very much, Franc.", "103212": "I've had trouble installing mxnet in the first place for OSX Capitan. Full version of my question at StackOverflow: http://stackoverflow.com/questions/34525112/trouble-installing-mxnet-on-osx-capitan. \r\n\r\nThe short story is that when I try to build mxnet on OSX Capitan I'm getting an error in the file `/mxnet/dmlc-core/include/dmlc/base.h` associated with `sys/types.h` which is not found. I've verified that I have this file in my c++ library and have tried manually recoding the path reference in `base.h` but only created more errors.\r\n\r\nAny thoughts?", "103236": "Hi, guys~~\r\n\r\nJust a remainder. If you met any problems or questions, please feel free to open an issue on github.\r\n\r\nhttps://github.com/dmlc/mxnet/issues\r\n\r\nYou will get much much quicker and better response.\r\n\r\nQiang Kou", "103267": "The fix to the problem I was having was to update the ADD_CFLAGS variable in the `make/osx.mk file`. On my machine this looked like: \r\n\r\n`ADD_CFLAGS = -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -mmacosx-version-min=10.11`\r\n\r\nhttps://github.com/dmlc/mxnet/issues/1111", "103331": "Qiang\r\n\r\nI'm having the same issue that @Konrad Banachewicz posted, where \r\n \r\n data_train <- mx.io.CSVIter(\r\n data.csv = \"./train-64x64-data.csv\", data.shape = c(64, 64, 30),\r\n label.csv = \"./train-stytole.csv\", label.shape = 600,\r\n batch.size = batch_size\r\n )\r\n\r\nProduces the error: \r\n\r\n [16:29:28] ./dmlc-core/include/dmlc/logging.h:208: [16:29:28] src/data.cc:43: Unknown data type csv\r\n Error: [16:29:28] src/data.cc:43: Unknown data type csv\r\n\r\nI've successfully compiled the `mxnet` and built the R package on OSX Capitan. I am using `clang-omp` as my CC export and `clang-omp++` as my CXX export. Could this be causing my problem? \r\n\r\n(Quick update: I just compiled `mxnet` today so it should be running the most current version. It wasn't clear to me whether the patch Qiang mentioned above for Windows was meant to apply here as well. `installpackages(mxnet)` method failed for me, so I built the package using option 2, from the command line, within the `mxnet` root folder)", "103829": "Hi Qiang, I finally set everything up on my end and have run the mxnet scripts in both R and python. The only modifications I made to the code were to change mx.gpu to mx.cpu (I don't have an Invidia graphics card). I upped the number of training iterations from 65 to 100. Only as a point of information, the score I got with the code in R was ~0.08, while with python it was ~0.0385. I'll keep playing with it, but so far it looks like the Python code produces by far the best result.", "103831": "[quote=Aaron Polhamus;103829]\r\n\r\nHi Qiang, I finally set everything up on my end and have run the mxnet scripts in both R and python. The only modifications I made to the code were to change mx.gpu to mx.cpu (I don't have an Invidia graphics card). I upped the number of training iterations from 65 to 100. Only as a point of information, the score I got with the code in R was ~0.08, while with python it was ~0.0385. I'll keep playing with it, but so far it looks like the Python code produces by far the best result. \r\n\r\n[/quote]\r\nHi Aaron, how long does it take for your R code to run using cpu?\r\n\r\nThanks!", "103834": "[quote=Aaron Polhamus;103829]\r\n\r\nHi Qiang, I finally set everything up on my end and have run the mxnet scripts in both R and python. The only modifications I made to the code were to change mx.gpu to mx.cpu (I don't have an Invidia graphics card). I upped the number of training iterations from 65 to 100. Only as a point of information, the score I got with the code in R was ~0.08, while with python it was ~0.0385. I'll keep playing with it, but so far it looks like the Python code produces by far the best result. \r\n\r\n[/quote]\r\n\r\nIf the results are different, there might be something wrong.\r\n\r\nLet me double check.", "103839": "I'm on an OSX quad core with 2.6 GHz Intel Core i7 CPUs. Total run time for 100 training iterations is ~6 - 8 hrs per model (so 12 - 16 for both systole and diastole).", "103881": "I have somewhat inline results with what is mentioned in the readme with the python script (~0.042 with no seed defined so I won't be able to reproduce) but not the R script (>0.20) which also seems to error on a cold start. Library built less than a week ago on yosemite. I will look into it and post updates to github issues.", "103882": "FWIW: i had a similar experience with the R script (score > 0.2). I am curious if anybody replicated the Python result in R on a cpu.", "104186": "Hi,\r\n\r\ni have run `Train.R` ,i got this error\r\n\r\n**Error:** \r\n[10:24:36] include/dmlc/logging.h:208: [10:24:36] src/io/local_filesys.cc:61: LocalFileSystem.GetPathInfo ./train-stytole.csv Error:No such file or directory\r\nError: [10:24:36] src/io/local_filesys.cc:61: LocalFileSystem.GetPathInfo ./train-stytole.csv Error:No such file or directory\r\n\r\nThis error is because of two files(`train-stytole.csv` , `train-diastole.csv`) are not generated by `Preprocessing.py`.\r\ncan anyone help me to generating these files\r\n 1) `train-stytole.csv` 2) `train-diastole.csv` ,or how to fix this error. thanks in advance.", "104187": "Those files are generated by Train.py\r\n\r\n def encode_label(label_data):\r\n systole = label_data[:, 1]\r\n diastole = label_data[:, 2]\r\n systole_encode = np.array([\r\n (x < np.arange(600)) for x in systole\r\n ], dtype=np.uint8)\r\n diastole_encode = np.array([\r\n (x < np.arange(600)) for x in diastole\r\n ], dtype=np.uint8)\r\n return systole_encode, diastole_encode\r\n \r\n def encode_csv(label_csv, systole_csv, diastole_csv):\r\n systole_encode, diastole_encode = encode_label(np.loadtxt(label_csv, delimiter=\",\"))\r\n np.savetxt(systole_csv, systole_encode, delimiter=\",\", fmt=\"%g\")\r\n np.savetxt(diastole_csv, diastole_encode, delimiter=\",\", fmt=\"%g\")\r\n \r\n encode_csv(\"./train-label.csv\", \"./train-systole.csv\", \"./train-diastole.csv\")\r\n\r\nI fixed the typo in \"stytole\" so you might want to revert that.", "104626": "would be nice if we can there is some pkgs in r that does the preprocessing pipelines as well.", "104627": "The preprocessing could be done in R, since we can read the data format\r\n\r\nhttps://cran.r-project.org/web/packages/oro.dicom/index.html", "106671": "[quote=Qiang;104627]\r\n\r\nThe preprocessing could be done in R, since we can read the data format\r\n\r\nhttps://cran.r-project.org/web/packages/oro.dicom/index.html\r\n\r\n[/quote]\r\n\r\nThis seems to be nice, any r hackers who want to take a stab?", "120625": "I have been able to get this to work a few weeks ago, but now I'm getting an error running:\r\n\r\nnormed <- preproc.image(im, mean.img)\r\n\r\n Error: Expecting a four-dimensional array\r\n\r\nI had to previously convert the parrots.png file to parrots.bmp to get the process to work,\r\nbut now I get the same error when using the png file.\r\n\r\nThank you.", "120642": "[quote=tjclifford;120625]\r\n\r\nI have been able to get this to work a few weeks ago, but now I'm getting an error running:\r\n\r\nnormed <- preproc.image(im, mean.img)\r\n\r\n Error: Expecting a four-dimensional array\r\n\r\nI had to previously convert the parrots.png file to parrots.bmp to get the process to work,\r\nbut now I get the same error when using the png file.\r\n\r\nThank you.\r\n\r\n[/quote]\r\n\r\nYou met this error because `imager` changed their APIs.\r\n\r\nWe have also updated the code to follow their changes, please see the latest documents: http://mxnet.dmlc.ml/en/latest/packages/r/classifyRealImageWithPretrainedModel.html", "125615": "Hey,\r\n\r\ncan't find any help so i'm posting this here, hoping some of you'll be able to enlight my day!\r\nhttps://github.com/dmlc/mxnet/issues/2535\r\n\r\nJacques,", "132144": "> You met this error because imager changed their APIs.\r\n> \r\n> We have also updated the code to follow their changes, please see the\r\n> latest documents:\r\n> http://mxnet.dmlc.ml/en/latest/packages/r/classifyRealImageWithPretrainedModel.html\r\n\r\n\r\nThis link no longer works and I really hope to try this out. Is there a new link with the fix to the imager API? Thanks!", "161653": "Good news for R users!" }, "source": "meta" }