{ "id": 2485, "title": "\"cap_and_update_priors\" in the benchmark code", "url": "/competitions/predict-closed-questions-on-stack-overflow/discussion/2485", "author_name": "", "post_date": "2012-08-28T21:53:15.403Z", "votes": null, "comment_count": 7, "views": 2837, "content": "

At first glance, the random forest benchmark seems very straightforward:

\r\n
\r\n
    rf.fit(fea,\r\ndata["OpenStatus"])
\r\n
    data \r\n= cu.get_dataframe(test_file)
\r\n
    test_features \r\n= features.extract_features(feature_names,\r\ndata)
\r\n
    probs \r\n= rf.predict_proba(test_features)
\r\n

\r\n
\r\n
\r\n
However, my own random forest (implemented in R) has nowhere near the same performance, and this performance gap is probably due to this piece of code:
\r\n
\r\n
\r\n

\r\n    print("Calculating priors and updating posteriors")
\r\n
    new_priors \r\n= cu.get_priors(full_train_file)
\r\n
    old_priors \r\n= cu.get_priors(train_file)
\r\n
    probs \r\n= cu.cap_and_update_priors(old_priors,\r\nprobs, \r\nnew_priors, 0.001)
\r\n
\r\n
I'm not very familiar with python.  Could someone please explain what this block of code does, and perhaps help me implement it in R?
\r\n
    
\r\n
", "messages": [ { "id": "13579", "postDate": "08/28/2012 21:53:15", "content": "

At first glance, the random forest benchmark seems very straightforward:

\r\n
\r\n
    rf.fit(fea,\r\ndata["OpenStatus"])
\r\n
    data \r\n= cu.get_dataframe(test_file)
\r\n
    test_features \r\n= features.extract_features(feature_names,\r\ndata)
\r\n
    probs \r\n= rf.predict_proba(test_features)
\r\n

\r\n
\r\n
\r\n
However, my own random forest (implemented in R) has nowhere near the same performance, and this performance gap is probably due to this piece of code:
\r\n
\r\n
\r\n

\r\n    print("Calculating priors and updating posteriors")
\r\n
    new_priors \r\n= cu.get_priors(full_train_file)
\r\n
    old_priors \r\n= cu.get_priors(train_file)
\r\n
    probs \r\n= cu.cap_and_update_priors(old_priors,\r\nprobs, \r\nnew_priors, 0.001)
\r\n
\r\n
I'm not very familiar with python.  Could someone please explain what this block of code does, and perhaps help me implement it in R?
\r\n
    
\r\n
", "rawMarkdown": "", "votes": null }, { "id": "13583", "postDate": "08/28/2012 23:15:48", "content": "

The first line of basic_benchmark is this:

\r\n
import competition_utilities as cu
\r\n

which means cu.xyz will be found in competition_utilities.py as def xyz:

", "rawMarkdown": "", "votes": null }, { "id": "13602", "postDate": "08/29/2012 12:29:11", "content": "

Right, I get that.  The relevant functions seem to be "get_priors," "update_prior," and "cap_and_update_priors."  I'm trying to understand what these functions are actually doing.  For example, what is the purpose of these lines?

\r\n
old_priors \r\n= np.kron(np.ones((np.size(old_posteriors,\r\n0), \r\n1)), old_priors)
\r\n
new_priors \r\n= np.kron(np.ones((np.size(old_posteriors,\r\n0), \r\n1)), new_priors)
\r\n


\r\n

", "rawMarkdown": "", "votes": null }, { "id": "13623", "postDate": "08/29/2012 19:40:29", "content": "

Here is my attempt to answer my own question, and port the competition utils to R

\r\n
\r\n

cap_predictions <- function(probs, epsilon=0.001){
  probs[probs>1-epsilon] = 1-epsilon
  probs[probs<epsilon] = epsilon
  probs = probs / rowSums(probs)
  return(probs)
}

get_priors <- function(file_name){
  closed_reasons = read.csv(file_name,
    colClasses=c(rep("NULL", 14), "character"), header = TRUE)[,1]
  closed_reason_counts = table(closed_reasons)
  total = length(closed_reasons)
  priors = closed_reason_counts/total
  return(priors)
}

update_priors <- function(old_prior,  old_posterior, new_prior){
  evidence_ratio = (old_prior*(1-old_posterior)) / 
    (old_posterior*(1-old_prior))
  new_posterior = new_prior / (new_prior + (1-new_prior)*evidence_ratio)
  return(new_posterior)
}

cap_and_update_priors <- function(old_priors, old_posteriors, 
                                  new_priors, epsilon=0.001){
  old_posteriors = cap_predictions(old_posteriors, epsilon)
  
  old_priors = kronecker(
    matrix(old_priors, ncol=length(old_priors)), 
    matrix(rep(1, nrow(old_posteriors))), nrow=nrow(old_posteriors))
  
  new_priors = kronecker(
    matrix(new_priors, ncol=length(new_priors)), 
    matrix(rep(1, nrow(old_posteriors))), nrow=nrow(old_posteriors))
  
  evidence_ratio = (old_priors*(1-old_posteriors)) / 
    (old_posteriors*(1-old_priors))
  
  new_posteriors = new_priors / (new_priors + (1-new_priors)*evidence_ratio)
  new_posteriors = cap_predictions(new_posteriors, epsilon)
  return(new_posteriors)
}
\r\n
", "rawMarkdown": "", "votes": null }, { "id": "13645", "postDate": "08/30/2012 15:02:27", "content": "

Here is a good article describing the procedure of updating priors:

\r\n

http://www.mpia-hd.mpg.de/Gaia/publications/probcomb_TN.pdf

\r\n

See 3 - Replacing prior information. Equation 12 (and Appendix A) is basically what the code in question does.

", "rawMarkdown": "", "votes": null }, { "id": "13650", "postDate": "08/30/2012 16:51:43", "content": "

Ok, its seems that the purpose of updating priors in this specific context is that you've built the model on a stratified sample, of about 50% open and 50% closed questions. However, in reality about 94% of questions are open, so you should adjust your predictions\r\n accordingly.

\r\n

Does this make sense?

", "rawMarkdown": "", "votes": null }, { "id": "13695", "postDate": "08/31/2012 13:58:58", "content": "

Yes, correct.

", "rawMarkdown": "", "votes": null }, { "id": "13697", "postDate": "08/31/2012 15:18:37", "content": "

Thank you. I had already asked about this in an earlier thread, but no one answered for some reason.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 13583, "author_name": "andysloane", "author_url": "", "post_date": "08/28/2012 23:15:48", "content": "

The first line of basic_benchmark is this:

\r\n
import competition_utilities as cu
\r\n

which means cu.xyz will be found in competition_utilities.py as def xyz:

", "votes": null, "replies": [] }, { "id": 13602, "author_name": "zachmayer", "author_url": "", "post_date": "08/29/2012 12:29:11", "content": "

Right, I get that.  The relevant functions seem to be "get_priors," "update_prior," and "cap_and_update_priors."  I'm trying to understand what these functions are actually doing.  For example, what is the purpose of these lines?

\r\n
old_priors \r\n= np.kron(np.ones((np.size(old_posteriors,\r\n0), \r\n1)), old_priors)
\r\n
new_priors \r\n= np.kron(np.ones((np.size(old_posteriors,\r\n0), \r\n1)), new_priors)
\r\n


\r\n

", "votes": null, "replies": [] }, { "id": 13623, "author_name": "zachmayer", "author_url": "", "post_date": "08/29/2012 19:40:29", "content": "

Here is my attempt to answer my own question, and port the competition utils to R

\r\n
\r\n

cap_predictions <- function(probs, epsilon=0.001){
  probs[probs>1-epsilon] = 1-epsilon
  probs[probs<epsilon] = epsilon
  probs = probs / rowSums(probs)
  return(probs)
}

get_priors <- function(file_name){
  closed_reasons = read.csv(file_name,
    colClasses=c(rep("NULL", 14), "character"), header = TRUE)[,1]
  closed_reason_counts = table(closed_reasons)
  total = length(closed_reasons)
  priors = closed_reason_counts/total
  return(priors)
}

update_priors <- function(old_prior,  old_posterior, new_prior){
  evidence_ratio = (old_prior*(1-old_posterior)) / 
    (old_posterior*(1-old_prior))
  new_posterior = new_prior / (new_prior + (1-new_prior)*evidence_ratio)
  return(new_posterior)
}

cap_and_update_priors <- function(old_priors, old_posteriors, 
                                  new_priors, epsilon=0.001){
  old_posteriors = cap_predictions(old_posteriors, epsilon)
  
  old_priors = kronecker(
    matrix(old_priors, ncol=length(old_priors)), 
    matrix(rep(1, nrow(old_posteriors))), nrow=nrow(old_posteriors))
  
  new_priors = kronecker(
    matrix(new_priors, ncol=length(new_priors)), 
    matrix(rep(1, nrow(old_posteriors))), nrow=nrow(old_posteriors))
  
  evidence_ratio = (old_priors*(1-old_posteriors)) / 
    (old_posteriors*(1-old_priors))
  
  new_posteriors = new_priors / (new_priors + (1-new_priors)*evidence_ratio)
  new_posteriors = cap_predictions(new_posteriors, epsilon)
  return(new_posteriors)
}
\r\n
", "votes": null, "replies": [] }, { "id": 13645, "author_name": "chisquared", "author_url": "", "post_date": "08/30/2012 15:02:27", "content": "

Here is a good article describing the procedure of updating priors:

\r\n

http://www.mpia-hd.mpg.de/Gaia/publications/probcomb_TN.pdf

\r\n

See 3 - Replacing prior information. Equation 12 (and Appendix A) is basically what the code in question does.

", "votes": null, "replies": [] }, { "id": 13650, "author_name": "zachmayer", "author_url": "", "post_date": "08/30/2012 16:51:43", "content": "

Ok, its seems that the purpose of updating priors in this specific context is that you've built the model on a stratified sample, of about 50% open and 50% closed questions. However, in reality about 94% of questions are open, so you should adjust your predictions\r\n accordingly.

\r\n

Does this make sense?

", "votes": null, "replies": [] }, { "id": 13695, "author_name": "chisquared", "author_url": "", "post_date": "08/31/2012 13:58:58", "content": "

Yes, correct.

", "votes": null, "replies": [] }, { "id": 13697, "author_name": "cjauvin", "author_url": "", "post_date": "08/31/2012 15:18:37", "content": "

Thank you. I had already asked about this in an earlier thread, but no one answered for some reason.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "13579": "", "13583": "", "13602": "", "13623": "", "13645": "", "13650": "", "13695": "", "13697": "" }, "source": "meta" }