I am using same input/output dimensions as the one used in Bing Xu's post. \nI transformed the label data to a list of arrays with length 600.
\n\nNow for my Caffe network I have a fully connected (inner product) layer with 600 output as my second to last layer, and a softmaxloss layer as my last layer.
\n\nWhen I try to train my network, I get an error saying that the dimension of my output and label do not match.
\n\n"softmax_loss_layer.cpp:42] Check failed: outer_num_ * inner_num_ == bottom[1]->count() (1 vs. 600) Number of labels must match number of predictions; e.g., if softmax axis == 1 and prediction shape is (N, C, H, W), label count (number of labels) must be N*H*W, with integer values in {0, 1, ..., C-1}."\nBut I don't really understand why that's the case. I have my label data stored in a lmdb database and the program seems to be able to detect that the size of my label is 600. But I don't understand why the prediction shape, or outer_num_ * inner_num_ seems to be 1, even though my last Fully Connected layer has 600 outputs...
\n\nAny help is appreciated!!!!!!
", "messages": [ { "id": "110524", "postDate": "03/06/2016 09:31:55", "content": "I am using same input/output dimensions as the one used in Bing Xu's post. \nI transformed the label data to a list of arrays with length 600.
\n\nNow for my Caffe network I have a fully connected (inner product) layer with 600 output as my second to last layer, and a softmaxloss layer as my last layer.
\n\nWhen I try to train my network, I get an error saying that the dimension of my output and label do not match.
\n\n"softmax_loss_layer.cpp:42] Check failed: outer_num_ * inner_num_ == bottom[1]->count() (1 vs. 600) Number of labels must match number of predictions; e.g., if softmax axis == 1 and prediction shape is (N, C, H, W), label count (number of labels) must be N*H*W, with integer values in {0, 1, ..., C-1}."\nBut I don't really understand why that's the case. I have my label data stored in a lmdb database and the program seems to be able to detect that the size of my label is 600. But I don't understand why the prediction shape, or outer_num_ * inner_num_ seems to be 1, even though my last Fully Connected layer has 600 outputs...
\n\nAny help is appreciated!!!!!!
", "rawMarkdown": "I am using same input/output dimensions as the one used in Bing Xu's post. \r\nI transformed the label data to a list of arrays with length 600. \r\n\r\nNow for my Caffe network I have a fully connected (inner product) layer with 600 output as my second to last layer, and a softmaxloss layer as my last layer. \r\n\r\nWhen I try to train my network, I get an error saying that the dimension of my output and label do not match. \r\n\r\n \"softmax_loss_layer.cpp:42] Check failed: outer_num_ * inner_num_ == bottom[1]->count() (1 vs. 600) Number of labels must match number of predictions; e.g., if softmax axis == 1 and prediction shape is (N, C, H, W), label count (number of labels) must be N*H*W, with integer values in {0, 1, ..., C-1}.\"\r\n\r\nBut I don't really understand why that's the case. I have my label data stored in a lmdb database and the program seems to be able to detect that the size of my label is 600. But I don't understand why the prediction shape, or outer_num_ * inner_num_ seems to be 1, even though my last Fully Connected layer has 600 outputs... \r\n\r\nAny help is appreciated!!!!!!", "votes": null }, { "id": "110546", "postDate": "03/06/2016 13:52:39", "content": "Can you share the last few layers of your train_val.prototxt, I.e. the fully-connected layer onwards?
", "rawMarkdown": "Can you share the last few layers of your train_val.prototxt, I.e. the fully-connected layer onwards?", "votes": null }, { "id": "110559", "postDate": "03/06/2016 17:25:22", "content": "layer {\n name: "fcc5"\n type: "InnerProduct"\n bottom: "pool4"\n top: "fcc5"\n inner_product_param {\n num_output: 1000\n }\n}\nlayer {\n name: "relu5"\n type: "ReLU"\n bottom: "fcc5"\n top: "fcc5"\n}\nlayer {\n name: "fcc6"\n type: "InnerProduct"\n bottom: "fcc5"\n top: "fcc6"\n inner_product_param {\n num_output: 600\n }\n}\nlayer {\n name: "loss"\n type: "SoftmaxWithLoss"\n bottom: "fcc6"\n bottom: "label"\n top: "loss"\n loss_param {\n normalize: true\n }\n}\nThe lmdb database for the label data was created by first forming an array of arrays. So like there are couple thousand inner arrays, and each of those inner array has length 600. Then, since from sample scripts online it seems that caffe wants data to be in datum format and then be stored in lmdb database, I did something like this to create my label lmdb database:
\n\ndb_label_diastole = lmdb.open("db_label_diastole", map_size =1e12)\n\nwith db_label_systole.begin(write=True) as txn_img:\n for label in diastole_encode:\n # now label would be an array of 600 ints. But input to array_to_datum format should be 3D so we expand it:\n datum = caffe.io.array_to_datum(np.expand_dims(np.expand_dims(label, axis=1), axis=1))\n txn_img.put("{:0>10d}".format(diastole_count),datum.SerializeToString())\n diastole_count+=1\nLet me know where I did wrong! I also tried making the last fully connected layer to have only 2, or even 1 output, but I get the same error (with the same numbers too, ie:1 vs. 600)
", "rawMarkdown": "layer {\r\n name: \"fcc5\"\r\n type: \"InnerProduct\"\r\n bottom: \"pool4\"\r\n top: \"fcc5\"\r\n inner_product_param {\r\n num_output: 1000\r\n }\r\n }\r\n layer {\r\n name: \"relu5\"\r\n type: \"ReLU\"\r\n bottom: \"fcc5\"\r\n top: \"fcc5\"\r\n }\r\n layer {\r\n name: \"fcc6\"\r\n type: \"InnerProduct\"\r\n bottom: \"fcc5\"\r\n top: \"fcc6\"\r\n inner_product_param {\r\n num_output: 600\r\n }\r\n }\r\n layer {\r\n name: \"loss\"\r\n type: \"SoftmaxWithLoss\"\r\n bottom: \"fcc6\"\r\n bottom: \"label\"\r\n top: \"loss\"\r\n loss_param {\r\n normalize: true\r\n }\r\n }\r\n\r\n\r\nThe lmdb database for the label data was created by first forming an array of arrays. So like there are couple thousand inner arrays, and each of those inner array has length 600. Then, since from sample scripts online it seems that caffe wants data to be in datum format and then be stored in lmdb database, I did something like this to create my label lmdb database:\r\n\r\n db_label_diastole = lmdb.open(\"db_label_diastole\", map_size =1e12)\r\n \r\n with db_label_systole.begin(write=True) as txn_img:\r\n for label in diastole_encode:\r\n # now label would be an array of 600 ints. But input to array_to_datum format should be 3D so we expand it:\r\n datum = caffe.io.array_to_datum(np.expand_dims(np.expand_dims(label, axis=1), axis=1))\r\n txn_img.put(\"{:0>10d}\".format(diastole_count),datum.SerializeToString())\r\n diastole_count+=1\r\n\r\nLet me know where I did wrong! I also tried making the last fully connected layer to have only 2, or even 1 output, but I get the same error (with the same numbers too, ie:1 vs. 600)", "votes": null }, { "id": "110563", "postDate": "03/06/2016 17:53:56", "content": "I think your label datums have shape (batchsize,600,1,1) because you've used np.expand_dims twice with axis=1. You need to reshape them to (batchsize,600). Try adding the following layer before your softmax layer:
\n\nlayer {\n name: "label_reshape"\n type: "Reshape"\n bottom: "label"\n top: "label"\n reshape_param {\n shape {\n dim: 0\n dim: -1\n }\n }\n}
", "rawMarkdown": "I think your label datums have shape (batchsize,600,1,1) because you've used np.expand_dims twice with axis=1. You need to reshape them to (batchsize,600). Try adding the following layer before your softmax layer:\r\n\r\n layer {\r\n name: \"label_reshape\"\r\n type: \"Reshape\"\r\n bottom: \"label\"\r\n top: \"label\"\r\n reshape_param {\r\n shape {\r\n dim: 0\r\n dim: -1\r\n }\r\n }\r\n}", "votes": null }, { "id": "110573", "postDate": "03/06/2016 19:03:55", "content": "@Senecaur
\n\nThank you for your response. I tried the reshape layer but I don't think it's working. I even tried shape{ dim: 0 dim:600 dim:1 dim:-1} and shape{dim:0 dim:-1 dim:1 dim:1} but all of them give pretty much the same result. As caffe is creating the layers I can see the following printouts (with reshape layer of shape{dim:0 dim:-1}):
\n\nI0306 10:56:59.650076 2033205248 layer_factory.hpp:76] Creating layer label\nI0306 10:56:59.650144 2033205248 net.cpp:111] Creating Layer label\nI0306 10:56:59.650161 2033205248 net.cpp:434] label -> label\nI0306 10:56:59.651201 2138112 db_lmdb.cpp:22] Opened lmdb db_label_systole/\nI0306 10:56:59.651291 2033205248 data_layer.cpp:44] output data size: 1,600,1,1\nI0306 10:56:59.651337 2033205248 net.cpp:156] Setting up label\nI0306 10:56:59.651352 2033205248 net.cpp:164] Top shape: 1 600 1 1 (600)\nSo it seems that you are right, the label data was indeed stored in the shape of 1x600x1x1.
\n\nI0306 10:59:56.611135 2033205248 net.cpp:111] Creating Layer fcc6\nI0306 10:59:56.611140 2033205248 net.cpp:478] fcc6 <- fcc5\nI0306 10:59:56.611147 2033205248 net.cpp:434] fcc6 -> fcc6\nI0306 10:59:56.612571 2033205248 net.cpp:156] Setting up fcc6\nI0306 10:59:56.612597 2033205248 net.cpp:164] Top shape: 1 600 (600)\nI0306 10:59:56.612617 2033205248 layer_factory.hpp:76] Creating layer label_reshape\nI0306 10:59:56.612640 2033205248 net.cpp:111] Creating Layer label_reshape\nI0306 10:59:56.612651 2033205248 net.cpp:478] label_reshape <- label\nI0306 10:59:56.612663 2033205248 net.cpp:420] label_reshape -> label (in-place)\nI0306 10:59:56.612699 2033205248 net.cpp:156] Setting up label_reshape\nI0306 10:59:56.612709 2033205248 net.cpp:164] Top shape: 1 600 (600)\nSo it seems that label layer and the last fully connected layer has the same shape now. But how come when it gets to the loss layer, it still give me the "Check failed: outer_num_ * inner_num_ == bottom[1]->count() (1 vs. 600)" error? Somehow I feel like the loss layer is only outputing 1 output? even thought the fully connected layer before it has the shape of 1x600.
", "rawMarkdown": "Senecaur\r\n\r\nThank you for your response. I tried the reshape layer but I don't think it's working. I even tried shape{ dim: 0 dim:600 dim:1 dim:-1} and shape{dim:0 dim:-1 dim:1 dim:1} but all of them give pretty much the same result. As caffe is creating the layers I can see the following printouts (with reshape layer of shape{dim:0 dim:-1}):\r\n\r\n I0306 10:56:59.650076 2033205248 layer_factory.hpp:76] Creating layer label\r\n I0306 10:56:59.650144 2033205248 net.cpp:111] Creating Layer label\r\n I0306 10:56:59.650161 2033205248 net.cpp:434] label -> label\r\n I0306 10:56:59.651201 2138112 db_lmdb.cpp:22] Opened lmdb db_label_systole/\r\n I0306 10:56:59.651291 2033205248 data_layer.cpp:44] output data size: 1,600,1,1\r\n I0306 10:56:59.651337 2033205248 net.cpp:156] Setting up label\r\n I0306 10:56:59.651352 2033205248 net.cpp:164] Top shape: 1 600 1 1 (600)\r\n\r\nSo it seems that you are right, the label data was indeed stored in the shape of 1x600x1x1. \r\n\r\n I0306 10:59:56.611135 2033205248 net.cpp:111] Creating Layer fcc6\r\n I0306 10:59:56.611140 2033205248 net.cpp:478] fcc6 <- fcc5\r\n I0306 10:59:56.611147 2033205248 net.cpp:434] fcc6 -> fcc6\r\n I0306 10:59:56.612571 2033205248 net.cpp:156] Setting up fcc6\r\n I0306 10:59:56.612597 2033205248 net.cpp:164] Top shape: 1 600 (600)\r\n I0306 10:59:56.612617 2033205248 layer_factory.hpp:76] Creating layer label_reshape\r\n I0306 10:59:56.612640 2033205248 net.cpp:111] Creating Layer label_reshape\r\n I0306 10:59:56.612651 2033205248 net.cpp:478] label_reshape <- label\r\n I0306 10:59:56.612663 2033205248 net.cpp:420] label_reshape -> label (in-place)\r\n I0306 10:59:56.612699 2033205248 net.cpp:156] Setting up label_reshape\r\n I0306 10:59:56.612709 2033205248 net.cpp:164] Top shape: 1 600 (600)\r\n\r\nSo it seems that label layer and the last fully connected layer has the same shape now. But how come when it gets to the loss layer, it still give me the \"Check failed: outer_num_ * inner_num_ == bottom[1]->count() (1 vs. 600)\" error? Somehow I feel like the loss layer is only outputing 1 output? even thought the fully connected layer before it has the shape of 1x600.", "votes": null }, { "id": "110574", "postDate": "03/06/2016 19:16:44", "content": "I just realized, you shouldn't be using a softmaxwithloss layer for this. SoftmaxWithLoss expects the label to be a single number. If you want to actually predict the CDF then you need to use something like EuclideanLoss. Sorry I didn't spot that before.
\n\nAlso, when you do the reshaping shape{dim: 0 dim: -1} is 2D, not 4D. It is flattening the last three dimensions. So the variants you tried are not doing the same thing.
", "rawMarkdown": "I just realized, you shouldn't be using a softmaxwithloss layer for this. SoftmaxWithLoss expects the label to be a single number. If you want to actually predict the CDF then you need to use something like EuclideanLoss. Sorry I didn't spot that before.\r\n\r\nAlso, when you do the reshaping shape{dim: 0 dim: -1} is 2D, not 4D. It is flattening the last three dimensions. So the variants you tried are not doing the same thing.", "votes": null }, { "id": "110576", "postDate": "03/06/2016 19:22:52", "content": "http://caffe.berkeleyvision.org/doxygen/classcaffe_1_1SoftmaxWithLossLayer.html
\n\nseems to suggest that the output of softmaxwithloss layer is (1x1x1x1).
\n\nand label should have dimension (Nx1x1x1) which in this case is (1x1x1x1)
\n\n....Should I use a different loss layer then? I don't understand why output of softmaxwithloss can only have one output. Bing Xu's MXNet setup used "return mx.symbol.LogisticRegressionOutput(data=fc1, name='softmax')" Isn't that equivalent to softmaxwithloss in caffe?
@Senecaur.
\n\nHaha I guess you answered as I was typing my new findings. Seems like we are agreeing on the same thing!\nBut Euclidean Loss doesn't seem to be the same as
\n\nmx.symbol.LogisticRegressionOutput(data=fc1, name='softmax'
or is it?
", "rawMarkdown": "Senecaur.\r\n\r\nHaha I guess you answered as I was typing my new findings. Seems like we are agreeing on the same thing!\r\nBut Euclidean Loss doesn't seem to be the same as \r\n\r\n`mx.symbol.LogisticRegressionOutput(data=fc1, name='softmax'` \r\n\r\nor is it?", "votes": null }, { "id": "110580", "postDate": "03/06/2016 19:41:41", "content": "LogisticRegressionOutput just produces the softmax vector, i.e. a 600-D vector of probabilities - it doesn't actually compute the loss. Bing Xu defined a custom loss function, CRPS to do that.
\n\nIn Caffe the SoftmaxWithLoss layer fuses these two steps together. CRPS is not immediately available in Caffe, so EuclideanLoss will not give the same result, but it is a reasonable thing to try.
", "rawMarkdown": "LogisticRegressionOutput just produces the softmax vector, i.e. a 600-D vector of probabilities - it doesn't actually compute the loss. Bing Xu defined a custom loss function, CRPS to do that. \r\n\r\nIn Caffe the SoftmaxWithLoss layer fuses these two steps together. CRPS is not immediately available in Caffe, so EuclideanLoss will not give the same result, but it is a reasonable thing to try.", "votes": null }, { "id": "110581", "postDate": "03/06/2016 19:52:17", "content": "Does Caffe have an option to pass in customized loss function, without having to create a new type of layer?
\n\nAlso New question..., when I change my loss layer type to EuclideanLoss, the program seems to get stuck at the creating data layer phase:
\n\nI0306 11:44:42.576844 2033205248 layer_factory.hpp:76] Creating layer data\nI0306 11:44:42.576877 2033205248 net.cpp:111] Creating Layer data\nI0306 11:44:42.576891 2033205248 net.cpp:434] data -> data\nAnd it just stays there without any further printouts. The only thing I changed was the type of the loss layer and I got rid of the reshape layer. How come the program would get stuck during the phase of creating data layer... before it even got to the loss layer?
\n\n[quote=Senecaur;110580]
\n\nLogisticRegressionOutput just produces the softmax vector, i.e. a 600-D vector of probabilities - it doesn't actually compute the loss. Bing Xu defined a custom loss function, CRPS to do that.
\n\nIn Caffe the SoftmaxWithLoss layer fuses these two steps together. CRPS is not immediately available in Caffe, so EuclideanLoss will not give the same result, but it is a reasonable thing to try.
\n\n[/quote]
", "rawMarkdown": "Does Caffe have an option to pass in customized loss function, without having to create a new type of layer? \r\n\r\nAlso New question..., when I change my loss layer type to EuclideanLoss, the program seems to get stuck at the creating data layer phase:\r\n\r\n I0306 11:44:42.576844 2033205248 layer_factory.hpp:76] Creating layer data\r\n I0306 11:44:42.576877 2033205248 net.cpp:111] Creating Layer data\r\n I0306 11:44:42.576891 2033205248 net.cpp:434] data -> data\r\n\r\nAnd it just stays there without any further printouts. The only thing I changed was the type of the loss layer and I got rid of the reshape layer. How come the program would get stuck during the phase of creating data layer... before it even got to the loss layer?\r\n\r\n\r\n[quote=Senecaur;110580]\r\n\r\nLogisticRegressionOutput just produces the softmax vector, i.e. a 600-D vector of probabilities - it doesn't actually compute the loss. Bing Xu defined a custom loss function, CRPS to do that. \r\n\r\nIn Caffe the SoftmaxWithLoss layer fuses these two steps together. CRPS is not immediately available in Caffe, so EuclideanLoss will not give the same result, but it is a reasonable thing to try.\r\n\r\n[/quote]", "votes": null }, { "id": "110583", "postDate": "03/06/2016 19:57:41", "content": "You have to create a new layer type or extend an existing one. If you are using the Caffe binaries then you'll have to actually edit the Caffe source, if you are using the Python API then you can define a loss layer through Python.
\n\nYou still want the reshape layer if you are going to use EuclideanLoss. I don't know why Caffe is hanging, check your prototxt file for errors I would guess.
", "rawMarkdown": "You have to create a new layer type or extend an existing one. If you are using the Caffe binaries then you'll have to actually edit the Caffe source, if you are using the Python API then you can define a loss layer through Python.\r\n\r\nYou still want the reshape layer if you are going to use EuclideanLoss. I don't know why Caffe is hanging, check your prototxt file for errors I would guess.", "votes": null }, { "id": "110585", "postDate": "03/06/2016 20:22:33", "content": "@Senecaur
\n\nI will definitely look into how to create my own loss layer through python later. I found couple links that seem helpful but do you have any recommended places to learn how to create my own loss layer?
\n\nBut for now, the program is still stuck at data layer creation.
\n\nlayer {\n name: "loss"\n type: "EuclideanLoss"\n bottom: "fcc6"\n bottom: "label"\n top: "loss"\n loss_param {\n normalize: true\n }\n}\nIf I change type to "SoftmaxWithLoss" then all the layers can get created with no problem except when I get to the loss layer I get the (1 vs 600) dimension error message.\nIf I change type to "EuclideanLoss" then the program gets stuck at the data layer creation, without ever getting to creating any of the other layers. Any idea why that might be the case? I did not change anything else.
\n\nThank you for your quick replies!
", "rawMarkdown": "Senecaur\r\n\r\nI will definitely look into how to create my own loss layer through python later. I found couple links that seem helpful but do you have any recommended places to learn how to create my own loss layer?\r\n\r\nBut for now, the program is still stuck at data layer creation. \r\n\r\n layer {\r\n name: \"loss\"\r\n type: \"EuclideanLoss\"\r\n bottom: \"fcc6\"\r\n bottom: \"label\"\r\n top: \"loss\"\r\n loss_param {\r\n normalize: true\r\n }\r\n }\r\nIf I change type to \"SoftmaxWithLoss\" then all the layers can get created with no problem except when I get to the loss layer I get the (1 vs 600) dimension error message.\r\nIf I change type to \"EuclideanLoss\" then the program gets stuck at the data layer creation, without ever getting to creating any of the other layers. Any idea why that might be the case? I did not change anything else.\r\n\r\nThank you for your quick replies!", "votes": null } ], "comments": [ { "id": 110546, "author_name": "senecaur", "author_url": "", "post_date": "03/06/2016 13:52:39", "content": "Can you share the last few layers of your train_val.prototxt, I.e. the fully-connected layer onwards?
", "votes": null, "replies": [] }, { "id": 110559, "author_name": "ilovesolder", "author_url": "", "post_date": "03/06/2016 17:25:22", "content": "layer {\n name: "fcc5"\n type: "InnerProduct"\n bottom: "pool4"\n top: "fcc5"\n inner_product_param {\n num_output: 1000\n }\n}\nlayer {\n name: "relu5"\n type: "ReLU"\n bottom: "fcc5"\n top: "fcc5"\n}\nlayer {\n name: "fcc6"\n type: "InnerProduct"\n bottom: "fcc5"\n top: "fcc6"\n inner_product_param {\n num_output: 600\n }\n}\nlayer {\n name: "loss"\n type: "SoftmaxWithLoss"\n bottom: "fcc6"\n bottom: "label"\n top: "loss"\n loss_param {\n normalize: true\n }\n}\nThe lmdb database for the label data was created by first forming an array of arrays. So like there are couple thousand inner arrays, and each of those inner array has length 600. Then, since from sample scripts online it seems that caffe wants data to be in datum format and then be stored in lmdb database, I did something like this to create my label lmdb database:
\n\ndb_label_diastole = lmdb.open("db_label_diastole", map_size =1e12)\n\nwith db_label_systole.begin(write=True) as txn_img:\n for label in diastole_encode:\n # now label would be an array of 600 ints. But input to array_to_datum format should be 3D so we expand it:\n datum = caffe.io.array_to_datum(np.expand_dims(np.expand_dims(label, axis=1), axis=1))\n txn_img.put("{:0>10d}".format(diastole_count),datum.SerializeToString())\n diastole_count+=1\nLet me know where I did wrong! I also tried making the last fully connected layer to have only 2, or even 1 output, but I get the same error (with the same numbers too, ie:1 vs. 600)
", "votes": null, "replies": [] }, { "id": 110563, "author_name": "senecaur", "author_url": "", "post_date": "03/06/2016 17:53:56", "content": "I think your label datums have shape (batchsize,600,1,1) because you've used np.expand_dims twice with axis=1. You need to reshape them to (batchsize,600). Try adding the following layer before your softmax layer:
\n\nlayer {\n name: "label_reshape"\n type: "Reshape"\n bottom: "label"\n top: "label"\n reshape_param {\n shape {\n dim: 0\n dim: -1\n }\n }\n}
", "votes": null, "replies": [] }, { "id": 110573, "author_name": "ilovesolder", "author_url": "", "post_date": "03/06/2016 19:03:55", "content": "@Senecaur
\n\nThank you for your response. I tried the reshape layer but I don't think it's working. I even tried shape{ dim: 0 dim:600 dim:1 dim:-1} and shape{dim:0 dim:-1 dim:1 dim:1} but all of them give pretty much the same result. As caffe is creating the layers I can see the following printouts (with reshape layer of shape{dim:0 dim:-1}):
\n\nI0306 10:56:59.650076 2033205248 layer_factory.hpp:76] Creating layer label\nI0306 10:56:59.650144 2033205248 net.cpp:111] Creating Layer label\nI0306 10:56:59.650161 2033205248 net.cpp:434] label -> label\nI0306 10:56:59.651201 2138112 db_lmdb.cpp:22] Opened lmdb db_label_systole/\nI0306 10:56:59.651291 2033205248 data_layer.cpp:44] output data size: 1,600,1,1\nI0306 10:56:59.651337 2033205248 net.cpp:156] Setting up label\nI0306 10:56:59.651352 2033205248 net.cpp:164] Top shape: 1 600 1 1 (600)\nSo it seems that you are right, the label data was indeed stored in the shape of 1x600x1x1.
\n\nI0306 10:59:56.611135 2033205248 net.cpp:111] Creating Layer fcc6\nI0306 10:59:56.611140 2033205248 net.cpp:478] fcc6 <- fcc5\nI0306 10:59:56.611147 2033205248 net.cpp:434] fcc6 -> fcc6\nI0306 10:59:56.612571 2033205248 net.cpp:156] Setting up fcc6\nI0306 10:59:56.612597 2033205248 net.cpp:164] Top shape: 1 600 (600)\nI0306 10:59:56.612617 2033205248 layer_factory.hpp:76] Creating layer label_reshape\nI0306 10:59:56.612640 2033205248 net.cpp:111] Creating Layer label_reshape\nI0306 10:59:56.612651 2033205248 net.cpp:478] label_reshape <- label\nI0306 10:59:56.612663 2033205248 net.cpp:420] label_reshape -> label (in-place)\nI0306 10:59:56.612699 2033205248 net.cpp:156] Setting up label_reshape\nI0306 10:59:56.612709 2033205248 net.cpp:164] Top shape: 1 600 (600)\nSo it seems that label layer and the last fully connected layer has the same shape now. But how come when it gets to the loss layer, it still give me the "Check failed: outer_num_ * inner_num_ == bottom[1]->count() (1 vs. 600)" error? Somehow I feel like the loss layer is only outputing 1 output? even thought the fully connected layer before it has the shape of 1x600.
", "votes": null, "replies": [] }, { "id": 110574, "author_name": "senecaur", "author_url": "", "post_date": "03/06/2016 19:16:44", "content": "I just realized, you shouldn't be using a softmaxwithloss layer for this. SoftmaxWithLoss expects the label to be a single number. If you want to actually predict the CDF then you need to use something like EuclideanLoss. Sorry I didn't spot that before.
\n\nAlso, when you do the reshaping shape{dim: 0 dim: -1} is 2D, not 4D. It is flattening the last three dimensions. So the variants you tried are not doing the same thing.
", "votes": null, "replies": [] }, { "id": 110576, "author_name": "ilovesolder", "author_url": "", "post_date": "03/06/2016 19:22:52", "content": "http://caffe.berkeleyvision.org/doxygen/classcaffe_1_1SoftmaxWithLossLayer.html
\n\nseems to suggest that the output of softmaxwithloss layer is (1x1x1x1).
\n\nand label should have dimension (Nx1x1x1) which in this case is (1x1x1x1)
\n\n....Should I use a different loss layer then? I don't understand why output of softmaxwithloss can only have one output. Bing Xu's MXNet setup used "return mx.symbol.LogisticRegressionOutput(data=fc1, name='softmax')" Isn't that equivalent to softmaxwithloss in caffe?
@Senecaur.
\n\nHaha I guess you answered as I was typing my new findings. Seems like we are agreeing on the same thing!\nBut Euclidean Loss doesn't seem to be the same as
\n\nmx.symbol.LogisticRegressionOutput(data=fc1, name='softmax'
or is it?
", "votes": null, "replies": [] }, { "id": 110580, "author_name": "senecaur", "author_url": "", "post_date": "03/06/2016 19:41:41", "content": "LogisticRegressionOutput just produces the softmax vector, i.e. a 600-D vector of probabilities - it doesn't actually compute the loss. Bing Xu defined a custom loss function, CRPS to do that.
\n\nIn Caffe the SoftmaxWithLoss layer fuses these two steps together. CRPS is not immediately available in Caffe, so EuclideanLoss will not give the same result, but it is a reasonable thing to try.
", "votes": null, "replies": [] }, { "id": 110581, "author_name": "ilovesolder", "author_url": "", "post_date": "03/06/2016 19:52:17", "content": "Does Caffe have an option to pass in customized loss function, without having to create a new type of layer?
\n\nAlso New question..., when I change my loss layer type to EuclideanLoss, the program seems to get stuck at the creating data layer phase:
\n\nI0306 11:44:42.576844 2033205248 layer_factory.hpp:76] Creating layer data\nI0306 11:44:42.576877 2033205248 net.cpp:111] Creating Layer data\nI0306 11:44:42.576891 2033205248 net.cpp:434] data -> data\nAnd it just stays there without any further printouts. The only thing I changed was the type of the loss layer and I got rid of the reshape layer. How come the program would get stuck during the phase of creating data layer... before it even got to the loss layer?
\n\n[quote=Senecaur;110580]
\n\nLogisticRegressionOutput just produces the softmax vector, i.e. a 600-D vector of probabilities - it doesn't actually compute the loss. Bing Xu defined a custom loss function, CRPS to do that.
\n\nIn Caffe the SoftmaxWithLoss layer fuses these two steps together. CRPS is not immediately available in Caffe, so EuclideanLoss will not give the same result, but it is a reasonable thing to try.
\n\n[/quote]
", "votes": null, "replies": [] }, { "id": 110583, "author_name": "senecaur", "author_url": "", "post_date": "03/06/2016 19:57:41", "content": "You have to create a new layer type or extend an existing one. If you are using the Caffe binaries then you'll have to actually edit the Caffe source, if you are using the Python API then you can define a loss layer through Python.
\n\nYou still want the reshape layer if you are going to use EuclideanLoss. I don't know why Caffe is hanging, check your prototxt file for errors I would guess.
", "votes": null, "replies": [] }, { "id": 110585, "author_name": "ilovesolder", "author_url": "", "post_date": "03/06/2016 20:22:33", "content": "@Senecaur
\n\nI will definitely look into how to create my own loss layer through python later. I found couple links that seem helpful but do you have any recommended places to learn how to create my own loss layer?
\n\nBut for now, the program is still stuck at data layer creation.
\n\nlayer {\n name: "loss"\n type: "EuclideanLoss"\n bottom: "fcc6"\n bottom: "label"\n top: "loss"\n loss_param {\n normalize: true\n }\n}\nIf I change type to "SoftmaxWithLoss" then all the layers can get created with no problem except when I get to the loss layer I get the (1 vs 600) dimension error message.\nIf I change type to "EuclideanLoss" then the program gets stuck at the data layer creation, without ever getting to creating any of the other layers. Any idea why that might be the case? I did not change anything else.
\n\nThank you for your quick replies!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "110524": "I am using same input/output dimensions as the one used in Bing Xu's post. \r\nI transformed the label data to a list of arrays with length 600. \r\n\r\nNow for my Caffe network I have a fully connected (inner product) layer with 600 output as my second to last layer, and a softmaxloss layer as my last layer. \r\n\r\nWhen I try to train my network, I get an error saying that the dimension of my output and label do not match. \r\n\r\n \"softmax_loss_layer.cpp:42] Check failed: outer_num_ * inner_num_ == bottom[1]->count() (1 vs. 600) Number of labels must match number of predictions; e.g., if softmax axis == 1 and prediction shape is (N, C, H, W), label count (number of labels) must be N*H*W, with integer values in {0, 1, ..., C-1}.\"\r\n\r\nBut I don't really understand why that's the case. I have my label data stored in a lmdb database and the program seems to be able to detect that the size of my label is 600. But I don't understand why the prediction shape, or outer_num_ * inner_num_ seems to be 1, even though my last Fully Connected layer has 600 outputs... \r\n\r\nAny help is appreciated!!!!!!", "110546": "Can you share the last few layers of your train_val.prototxt, I.e. the fully-connected layer onwards?", "110559": "layer {\r\n name: \"fcc5\"\r\n type: \"InnerProduct\"\r\n bottom: \"pool4\"\r\n top: \"fcc5\"\r\n inner_product_param {\r\n num_output: 1000\r\n }\r\n }\r\n layer {\r\n name: \"relu5\"\r\n type: \"ReLU\"\r\n bottom: \"fcc5\"\r\n top: \"fcc5\"\r\n }\r\n layer {\r\n name: \"fcc6\"\r\n type: \"InnerProduct\"\r\n bottom: \"fcc5\"\r\n top: \"fcc6\"\r\n inner_product_param {\r\n num_output: 600\r\n }\r\n }\r\n layer {\r\n name: \"loss\"\r\n type: \"SoftmaxWithLoss\"\r\n bottom: \"fcc6\"\r\n bottom: \"label\"\r\n top: \"loss\"\r\n loss_param {\r\n normalize: true\r\n }\r\n }\r\n\r\n\r\nThe lmdb database for the label data was created by first forming an array of arrays. So like there are couple thousand inner arrays, and each of those inner array has length 600. Then, since from sample scripts online it seems that caffe wants data to be in datum format and then be stored in lmdb database, I did something like this to create my label lmdb database:\r\n\r\n db_label_diastole = lmdb.open(\"db_label_diastole\", map_size =1e12)\r\n \r\n with db_label_systole.begin(write=True) as txn_img:\r\n for label in diastole_encode:\r\n # now label would be an array of 600 ints. But input to array_to_datum format should be 3D so we expand it:\r\n datum = caffe.io.array_to_datum(np.expand_dims(np.expand_dims(label, axis=1), axis=1))\r\n txn_img.put(\"{:0>10d}\".format(diastole_count),datum.SerializeToString())\r\n diastole_count+=1\r\n\r\nLet me know where I did wrong! I also tried making the last fully connected layer to have only 2, or even 1 output, but I get the same error (with the same numbers too, ie:1 vs. 600)", "110563": "I think your label datums have shape (batchsize,600,1,1) because you've used np.expand_dims twice with axis=1. You need to reshape them to (batchsize,600). Try adding the following layer before your softmax layer:\r\n\r\n layer {\r\n name: \"label_reshape\"\r\n type: \"Reshape\"\r\n bottom: \"label\"\r\n top: \"label\"\r\n reshape_param {\r\n shape {\r\n dim: 0\r\n dim: -1\r\n }\r\n }\r\n}", "110573": "Senecaur\r\n\r\nThank you for your response. I tried the reshape layer but I don't think it's working. I even tried shape{ dim: 0 dim:600 dim:1 dim:-1} and shape{dim:0 dim:-1 dim:1 dim:1} but all of them give pretty much the same result. As caffe is creating the layers I can see the following printouts (with reshape layer of shape{dim:0 dim:-1}):\r\n\r\n I0306 10:56:59.650076 2033205248 layer_factory.hpp:76] Creating layer label\r\n I0306 10:56:59.650144 2033205248 net.cpp:111] Creating Layer label\r\n I0306 10:56:59.650161 2033205248 net.cpp:434] label -> label\r\n I0306 10:56:59.651201 2138112 db_lmdb.cpp:22] Opened lmdb db_label_systole/\r\n I0306 10:56:59.651291 2033205248 data_layer.cpp:44] output data size: 1,600,1,1\r\n I0306 10:56:59.651337 2033205248 net.cpp:156] Setting up label\r\n I0306 10:56:59.651352 2033205248 net.cpp:164] Top shape: 1 600 1 1 (600)\r\n\r\nSo it seems that you are right, the label data was indeed stored in the shape of 1x600x1x1. \r\n\r\n I0306 10:59:56.611135 2033205248 net.cpp:111] Creating Layer fcc6\r\n I0306 10:59:56.611140 2033205248 net.cpp:478] fcc6 <- fcc5\r\n I0306 10:59:56.611147 2033205248 net.cpp:434] fcc6 -> fcc6\r\n I0306 10:59:56.612571 2033205248 net.cpp:156] Setting up fcc6\r\n I0306 10:59:56.612597 2033205248 net.cpp:164] Top shape: 1 600 (600)\r\n I0306 10:59:56.612617 2033205248 layer_factory.hpp:76] Creating layer label_reshape\r\n I0306 10:59:56.612640 2033205248 net.cpp:111] Creating Layer label_reshape\r\n I0306 10:59:56.612651 2033205248 net.cpp:478] label_reshape <- label\r\n I0306 10:59:56.612663 2033205248 net.cpp:420] label_reshape -> label (in-place)\r\n I0306 10:59:56.612699 2033205248 net.cpp:156] Setting up label_reshape\r\n I0306 10:59:56.612709 2033205248 net.cpp:164] Top shape: 1 600 (600)\r\n\r\nSo it seems that label layer and the last fully connected layer has the same shape now. But how come when it gets to the loss layer, it still give me the \"Check failed: outer_num_ * inner_num_ == bottom[1]->count() (1 vs. 600)\" error? Somehow I feel like the loss layer is only outputing 1 output? even thought the fully connected layer before it has the shape of 1x600.", "110574": "I just realized, you shouldn't be using a softmaxwithloss layer for this. SoftmaxWithLoss expects the label to be a single number. If you want to actually predict the CDF then you need to use something like EuclideanLoss. Sorry I didn't spot that before.\r\n\r\nAlso, when you do the reshaping shape{dim: 0 dim: -1} is 2D, not 4D. It is flattening the last three dimensions. So the variants you tried are not doing the same thing.", "110576": "http://caffe.berkeleyvision.org/doxygen/classcaffe_1_1SoftmaxWithLossLayer.html\r\n\r\nseems to suggest that the output of softmaxwithloss layer is (1x1x1x1). \r\n\r\nand label should have dimension (Nx1x1x1) which in this case is (1x1x1x1)\r\n\r\n....Should I use a different loss layer then? I don't understand why output of softmaxwithloss can only have one output. Bing Xu's MXNet setup used `\"return mx.symbol.LogisticRegressionOutput(data=fc1, name='softmax')\"` Isn't that equivalent to softmaxwithloss in caffe?", "110577": "Senecaur.\r\n\r\nHaha I guess you answered as I was typing my new findings. Seems like we are agreeing on the same thing!\r\nBut Euclidean Loss doesn't seem to be the same as \r\n\r\n`mx.symbol.LogisticRegressionOutput(data=fc1, name='softmax'` \r\n\r\nor is it?", "110580": "LogisticRegressionOutput just produces the softmax vector, i.e. a 600-D vector of probabilities - it doesn't actually compute the loss. Bing Xu defined a custom loss function, CRPS to do that. \r\n\r\nIn Caffe the SoftmaxWithLoss layer fuses these two steps together. CRPS is not immediately available in Caffe, so EuclideanLoss will not give the same result, but it is a reasonable thing to try.", "110581": "Does Caffe have an option to pass in customized loss function, without having to create a new type of layer? \r\n\r\nAlso New question..., when I change my loss layer type to EuclideanLoss, the program seems to get stuck at the creating data layer phase:\r\n\r\n I0306 11:44:42.576844 2033205248 layer_factory.hpp:76] Creating layer data\r\n I0306 11:44:42.576877 2033205248 net.cpp:111] Creating Layer data\r\n I0306 11:44:42.576891 2033205248 net.cpp:434] data -> data\r\n\r\nAnd it just stays there without any further printouts. The only thing I changed was the type of the loss layer and I got rid of the reshape layer. How come the program would get stuck during the phase of creating data layer... before it even got to the loss layer?\r\n\r\n\r\n[quote=Senecaur;110580]\r\n\r\nLogisticRegressionOutput just produces the softmax vector, i.e. a 600-D vector of probabilities - it doesn't actually compute the loss. Bing Xu defined a custom loss function, CRPS to do that. \r\n\r\nIn Caffe the SoftmaxWithLoss layer fuses these two steps together. CRPS is not immediately available in Caffe, so EuclideanLoss will not give the same result, but it is a reasonable thing to try.\r\n\r\n[/quote]", "110583": "You have to create a new layer type or extend an existing one. If you are using the Caffe binaries then you'll have to actually edit the Caffe source, if you are using the Python API then you can define a loss layer through Python.\r\n\r\nYou still want the reshape layer if you are going to use EuclideanLoss. I don't know why Caffe is hanging, check your prototxt file for errors I would guess.", "110585": "Senecaur\r\n\r\nI will definitely look into how to create my own loss layer through python later. I found couple links that seem helpful but do you have any recommended places to learn how to create my own loss layer?\r\n\r\nBut for now, the program is still stuck at data layer creation. \r\n\r\n layer {\r\n name: \"loss\"\r\n type: \"EuclideanLoss\"\r\n bottom: \"fcc6\"\r\n bottom: \"label\"\r\n top: \"loss\"\r\n loss_param {\r\n normalize: true\r\n }\r\n }\r\nIf I change type to \"SoftmaxWithLoss\" then all the layers can get created with no problem except when I get to the loss layer I get the (1 vs 600) dimension error message.\r\nIf I change type to \"EuclideanLoss\" then the program gets stuck at the data layer creation, without ever getting to creating any of the other layers. Any idea why that might be the case? I did not change anything else.\r\n\r\nThank you for your quick replies!" }, "source": "meta" }