{ "id": 15915, "title": "Debugging Graphlab Create Optimizers", "url": "/competitions/dato-native/discussion/15915", "author_name": "", "post_date": "2015-08-12T21:27:18.220Z", "votes": null, "comment_count": 7, "views": 1108, "content": "

I attempted to fit a logistic regression classifier on the entire training data set, but ended up with a warning during training:

\n\n
\n

PROGRESS: Logistic regression:

\n \n

PROGRESS: --------------------------------------------------------

\n \n

PROGRESS: Number of examples : 96052

\n \n

PROGRESS: Number of classes : 2

\n \n

PROGRESS: Number of feature columns : 2

\n \n

PROGRESS: Number of unpacked features : 1781200

\n \n

PROGRESS: Number of coefficients : 1781201

\n \n

PROGRESS: Starting L-BFGS

\n \n

PROGRESS: --------------------------------------------------------

\n \n

PROGRESS:\n +-----------+----------+-----------+--------------+-------------------+---------------+

\n \n

PROGRESS: | Iteration | Passes | Step size | Elapsed Time |\n Training-accuracy |

\n \n

Validation-accuracy |

\n \n

PROGRESS:\n +---+----+--------------+---------------+-------------+-------------+

\n \n

PROGRESS: | 1 | 7 | 0.000001 | 7.679090 |\n 0.746294 | 0.621167

\n \n

|

\n \n

PROGRESS: Warning: Unusual termination criterion reached.

\n \n

Returning the best step found so far.

\n \n

PROGRESS: Warning: Rounding errors prevent further progress.

\n \n

There may not be a step which satisfies the sufficient decrease and\n curvature conditions.

\n \n

Tolerances may be too small or dataset may be poorly scaled.

\n
\n\n

I ran the same test, on a subset of the data (bucket0), and it trained for 10 iterations yielding satisfactory results. Has anyone else encountered this issue? Please let me know also if there is a GraphLab-Create User Group where I can direct this specific question too (if this is not the appropriate forum to post)

", "messages": [ { "id": "89224", "postDate": "08/12/2015 21:27:18", "content": "

I attempted to fit a logistic regression classifier on the entire training data set, but ended up with a warning during training:

\n\n
\n

PROGRESS: Logistic regression:

\n \n

PROGRESS: --------------------------------------------------------

\n \n

PROGRESS: Number of examples : 96052

\n \n

PROGRESS: Number of classes : 2

\n \n

PROGRESS: Number of feature columns : 2

\n \n

PROGRESS: Number of unpacked features : 1781200

\n \n

PROGRESS: Number of coefficients : 1781201

\n \n

PROGRESS: Starting L-BFGS

\n \n

PROGRESS: --------------------------------------------------------

\n \n

PROGRESS:\n +-----------+----------+-----------+--------------+-------------------+---------------+

\n \n

PROGRESS: | Iteration | Passes | Step size | Elapsed Time |\n Training-accuracy |

\n \n

Validation-accuracy |

\n \n

PROGRESS:\n +---+----+--------------+---------------+-------------+-------------+

\n \n

PROGRESS: | 1 | 7 | 0.000001 | 7.679090 |\n 0.746294 | 0.621167

\n \n

|

\n \n

PROGRESS: Warning: Unusual termination criterion reached.

\n \n

Returning the best step found so far.

\n \n

PROGRESS: Warning: Rounding errors prevent further progress.

\n \n

There may not be a step which satisfies the sufficient decrease and\n curvature conditions.

\n \n

Tolerances may be too small or dataset may be poorly scaled.

\n
\n\n

I ran the same test, on a subset of the data (bucket0), and it trained for 10 iterations yielding satisfactory results. Has anyone else encountered this issue? Please let me know also if there is a GraphLab-Create User Group where I can direct this specific question too (if this is not the appropriate forum to post)

", "rawMarkdown": "I attempted to fit a logistic regression classifier on the entire training data set, but ended up with a warning during training: \r\n\r\n\r\n> PROGRESS: Logistic regression:\r\n> \r\n> PROGRESS: --------------------------------------------------------\r\n> \r\n> PROGRESS: Number of examples : 96052\r\n> \r\n> PROGRESS: Number of classes : 2\r\n> \r\n> PROGRESS: Number of feature columns : 2\r\n> \r\n> PROGRESS: Number of unpacked features : 1781200\r\n> \r\n> PROGRESS: Number of coefficients : 1781201\r\n> \r\n> PROGRESS: Starting L-BFGS\r\n> \r\n> PROGRESS: --------------------------------------------------------\r\n> \r\n> PROGRESS:\r\n> +-----------+----------+-----------+--------------+-------------------+---------------+\r\n> \r\n> PROGRESS: | Iteration | Passes | Step size | Elapsed Time |\r\n> Training-accuracy | \r\n> \r\n> Validation-accuracy |\r\n> \r\n> PROGRESS:\r\n> +---+----+--------------+---------------+-------------+-------------+\r\n> \r\n> PROGRESS: | 1 | 7 | 0.000001 | 7.679090 |\r\n> 0.746294 | 0.621167 \r\n> \r\n> |\r\n> \r\n> PROGRESS: Warning: Unusual termination criterion reached.\r\n> \r\n> Returning the best step found so far.\r\n> \r\n> PROGRESS: Warning: Rounding errors prevent further progress. \r\n> \r\n> There may not be a step which satisfies the sufficient decrease and\r\n> curvature conditions. \r\n> \r\n> Tolerances may be too small or dataset may be poorly scaled.\r\n\r\n\r\nI ran the same test, on a subset of the data (bucket0), and it trained for 10 iterations yielding satisfactory results. Has anyone else encountered this issue? Please let me know also if there is a GraphLab-Create User Group where I can direct this specific question too (if this is not the appropriate forum to post)", "votes": null }, { "id": "89240", "postDate": "08/12/2015 23:41:28", "content": "

Interesting that it worked perfectly with bucket0 only, how much RAM did it need for bucket0? ... and then.... how much for all the buckets?. How much RAM do you have?.

", "rawMarkdown": "Interesting that it worked perfectly with bucket0 only, how much RAM did it need for bucket0? ... and then.... how much for all the buckets?. How much RAM do you have?.", "votes": null }, { "id": "89265", "postDate": "08/13/2015 03:18:28", "content": "

It seems like you are heavily overfitting with 1781200 variables and only 96052 examples. The default regularizer of 0.01 probably isn't high enough so the problem is singular/not-well defined. Can you try increasing the regularization on the problem?

", "rawMarkdown": "It seems like you are heavily overfitting with 1781200 variables and only 96052 examples. The default regularizer of 0.01 probably isn't high enough so the problem is singular/not-well defined. Can you try increasing the regularization on the problem?", "votes": null }, { "id": "89317", "postDate": "08/13/2015 20:36:19", "content": "

Thanks for both replies.

\n\n

@NxGTR I swapped out an svm for the logistic classifier, and it ended up working on the whole data. Haven't debugged the logistic, but for now, I'm going to use an SVM.

\n\n

@Srikrishna That's a really good point regarding that how my dimensionality is larger than my number of features. I'll try regularizing more to make my weight matrix sparser for all those features. Do you know of other methods on reducing that dimensionality (maybe playing with the BOW tokenizer? PCA?)

", "rawMarkdown": "Thanks for both replies. \r\n\r\n@NxGTR I swapped out an svm for the logistic classifier, and it ended up working on the whole data. Haven't debugged the logistic, but for now, I'm going to use an SVM.\r\n\r\n@Srikrishna That's a really good point regarding that how my dimensionality is larger than my number of features. I'll try regularizing more to make my weight matrix sparser for all those features. Do you know of other methods on reducing that dimensionality (maybe playing with the BOW tokenizer? PCA?)", "votes": null }, { "id": "89318", "postDate": "08/13/2015 20:40:46", "content": "

@Ryan: I took a close look to see what's going on. Your error has to do with how our regularization is defined. We do not scale it with the number of examples i.e usually, the regularization value is defined to be scaled invariant. That's probably why the default value is too small.

", "rawMarkdown": "Ryan: I took a close look to see what's going on. Your error has to do with how our regularization is defined. We do not scale it with the number of examples i.e usually, the regularization value is defined to be scaled invariant. That's probably why the default value is too small.", "votes": null }, { "id": "89321", "postDate": "08/13/2015 21:17:54", "content": "

[quote=Srikrishna Sridhar;89318]

\n\n

@Ryan: I took a close look to see what's going on. Your error has to do with how our regularization is defined. We do not scale it with the number of examples i.e usually, the regularization value is defined to be scaled invariant. That's probably why the default value is too small.

\n\n

[/quote]

\n\n

That reads as if you both were a team. Are you?

", "rawMarkdown": "[quote=Srikrishna Sridhar;89318]\r\n\r\n@Ryan: **I took a close look to see what's going on**. Your error has to do with how **our** regularization is defined. **We** do not scale it with the number of examples i.e usually, the regularization value is defined to be scaled invariant. That's probably why the default value is too small.\r\n\r\n[/quote]\r\n\r\nThat reads as if you both were a team. Are you?", "votes": null }, { "id": "89323", "postDate": "08/13/2015 21:28:17", "content": "

@NXGTR: I am not working with Ryan as part of a team. I should have been clearer in my communication. I implemented the code for the classifier training in GraphLab create and I was trying to debug the error in the code. The "our" code was referring to GraphLab Create code. Again, I apologize if it looked as though I was collaborating.

", "rawMarkdown": "NXGTR: I am not working with Ryan as part of a team. I should have been clearer in my communication. I implemented the code for the classifier training in GraphLab create and I was trying to debug the error in the code. The \"our\" code was referring to GraphLab Create code. Again, I apologize if it looked as though I was collaborating.", "votes": null }, { "id": "89324", "postDate": "08/13/2015 21:31:38", "content": "

@Srikrishna Sridhar,

\n\n

No need to apologize, my bad. Keep with the good work.

", "rawMarkdown": "Srikrishna Sridhar,\r\n\r\nNo need to apologize, my bad. Keep with the good work.", "votes": null } ], "comments": [ { "id": 89240, "author_name": "carloshuertas", "author_url": "", "post_date": "08/12/2015 23:41:28", "content": "

Interesting that it worked perfectly with bucket0 only, how much RAM did it need for bucket0? ... and then.... how much for all the buckets?. How much RAM do you have?.

", "votes": null, "replies": [] }, { "id": 89265, "author_name": "srikrishnasridhar1", "author_url": "", "post_date": "08/13/2015 03:18:28", "content": "

It seems like you are heavily overfitting with 1781200 variables and only 96052 examples. The default regularizer of 0.01 probably isn't high enough so the problem is singular/not-well defined. Can you try increasing the regularization on the problem?

", "votes": null, "replies": [] }, { "id": 89317, "author_name": "ryanlouie", "author_url": "", "post_date": "08/13/2015 20:36:19", "content": "

Thanks for both replies.

\n\n

@NxGTR I swapped out an svm for the logistic classifier, and it ended up working on the whole data. Haven't debugged the logistic, but for now, I'm going to use an SVM.

\n\n

@Srikrishna That's a really good point regarding that how my dimensionality is larger than my number of features. I'll try regularizing more to make my weight matrix sparser for all those features. Do you know of other methods on reducing that dimensionality (maybe playing with the BOW tokenizer? PCA?)

", "votes": null, "replies": [] }, { "id": 89318, "author_name": "srikrishnasridhar1", "author_url": "", "post_date": "08/13/2015 20:40:46", "content": "

@Ryan: I took a close look to see what's going on. Your error has to do with how our regularization is defined. We do not scale it with the number of examples i.e usually, the regularization value is defined to be scaled invariant. That's probably why the default value is too small.

", "votes": null, "replies": [] }, { "id": 89321, "author_name": "carloshuertas", "author_url": "", "post_date": "08/13/2015 21:17:54", "content": "

[quote=Srikrishna Sridhar;89318]

\n\n

@Ryan: I took a close look to see what's going on. Your error has to do with how our regularization is defined. We do not scale it with the number of examples i.e usually, the regularization value is defined to be scaled invariant. That's probably why the default value is too small.

\n\n

[/quote]

\n\n

That reads as if you both were a team. Are you?

", "votes": null, "replies": [] }, { "id": 89323, "author_name": "srikrishnasridhar1", "author_url": "", "post_date": "08/13/2015 21:28:17", "content": "

@NXGTR: I am not working with Ryan as part of a team. I should have been clearer in my communication. I implemented the code for the classifier training in GraphLab create and I was trying to debug the error in the code. The "our" code was referring to GraphLab Create code. Again, I apologize if it looked as though I was collaborating.

", "votes": null, "replies": [] }, { "id": 89324, "author_name": "carloshuertas", "author_url": "", "post_date": "08/13/2015 21:31:38", "content": "

@Srikrishna Sridhar,

\n\n

No need to apologize, my bad. Keep with the good work.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "89224": "I attempted to fit a logistic regression classifier on the entire training data set, but ended up with a warning during training: \r\n\r\n\r\n> PROGRESS: Logistic regression:\r\n> \r\n> PROGRESS: --------------------------------------------------------\r\n> \r\n> PROGRESS: Number of examples : 96052\r\n> \r\n> PROGRESS: Number of classes : 2\r\n> \r\n> PROGRESS: Number of feature columns : 2\r\n> \r\n> PROGRESS: Number of unpacked features : 1781200\r\n> \r\n> PROGRESS: Number of coefficients : 1781201\r\n> \r\n> PROGRESS: Starting L-BFGS\r\n> \r\n> PROGRESS: --------------------------------------------------------\r\n> \r\n> PROGRESS:\r\n> +-----------+----------+-----------+--------------+-------------------+---------------+\r\n> \r\n> PROGRESS: | Iteration | Passes | Step size | Elapsed Time |\r\n> Training-accuracy | \r\n> \r\n> Validation-accuracy |\r\n> \r\n> PROGRESS:\r\n> +---+----+--------------+---------------+-------------+-------------+\r\n> \r\n> PROGRESS: | 1 | 7 | 0.000001 | 7.679090 |\r\n> 0.746294 | 0.621167 \r\n> \r\n> |\r\n> \r\n> PROGRESS: Warning: Unusual termination criterion reached.\r\n> \r\n> Returning the best step found so far.\r\n> \r\n> PROGRESS: Warning: Rounding errors prevent further progress. \r\n> \r\n> There may not be a step which satisfies the sufficient decrease and\r\n> curvature conditions. \r\n> \r\n> Tolerances may be too small or dataset may be poorly scaled.\r\n\r\n\r\nI ran the same test, on a subset of the data (bucket0), and it trained for 10 iterations yielding satisfactory results. Has anyone else encountered this issue? Please let me know also if there is a GraphLab-Create User Group where I can direct this specific question too (if this is not the appropriate forum to post)", "89240": "Interesting that it worked perfectly with bucket0 only, how much RAM did it need for bucket0? ... and then.... how much for all the buckets?. How much RAM do you have?.", "89265": "It seems like you are heavily overfitting with 1781200 variables and only 96052 examples. The default regularizer of 0.01 probably isn't high enough so the problem is singular/not-well defined. Can you try increasing the regularization on the problem?", "89317": "Thanks for both replies. \r\n\r\n@NxGTR I swapped out an svm for the logistic classifier, and it ended up working on the whole data. Haven't debugged the logistic, but for now, I'm going to use an SVM.\r\n\r\n@Srikrishna That's a really good point regarding that how my dimensionality is larger than my number of features. I'll try regularizing more to make my weight matrix sparser for all those features. Do you know of other methods on reducing that dimensionality (maybe playing with the BOW tokenizer? PCA?)", "89318": "Ryan: I took a close look to see what's going on. Your error has to do with how our regularization is defined. We do not scale it with the number of examples i.e usually, the regularization value is defined to be scaled invariant. That's probably why the default value is too small.", "89321": "[quote=Srikrishna Sridhar;89318]\r\n\r\n@Ryan: **I took a close look to see what's going on**. Your error has to do with how **our** regularization is defined. **We** do not scale it with the number of examples i.e usually, the regularization value is defined to be scaled invariant. That's probably why the default value is too small.\r\n\r\n[/quote]\r\n\r\nThat reads as if you both were a team. Are you?", "89323": "NXGTR: I am not working with Ryan as part of a team. I should have been clearer in my communication. I implemented the code for the classifier training in GraphLab create and I was trying to debug the error in the code. The \"our\" code was referring to GraphLab Create code. Again, I apologize if it looked as though I was collaborating.", "89324": "Srikrishna Sridhar,\r\n\r\nNo need to apologize, my bad. Keep with the good work." }, "source": "meta" }