{ "id": 12740, "title": "CV scores vs Leaderboard", "url": "/competitions/malware-classification/discussion/12740", "author_name": "", "post_date": "2015-03-09T10:17:42.853Z", "votes": null, "comment_count": 15, "views": 3361, "content": "

How does your local CV score compare against the public LB?

I'm asking because my CV scores are directionally correct but seem to be consistently higher (around +0.005) than the LB.

", "messages": [ { "id": "65767", "postDate": "03/09/2015 10:17:42", "content": "

How does your local CV score compare against the public LB?

I'm asking because my CV scores are directionally correct but seem to be consistently higher (around +0.005) than the LB.

", "rawMarkdown": "", "votes": null }, { "id": "65778", "postDate": "03/09/2015 14:14:39", "content": "

my 4-fold CV is about 0.0007~0.001 higher than LB

", "rawMarkdown": "", "votes": null }, { "id": "65810", "postDate": "03/09/2015 19:32:39", "content": "

@Stergios, 

\n

Mine (10-fold) CV score compared to LB is similar to your situation. 

", "rawMarkdown": "", "votes": null }, { "id": "65815", "postDate": "03/09/2015 21:28:04", "content": "

I'm getting 0.02315174 in a 5-fold cv, but my LB score is ~0.109...i'm not sure why =p

", "rawMarkdown": "", "votes": null }, { "id": "65817", "postDate": "03/09/2015 22:37:15", "content": "

Mine is also about 0.001 -- 0.002 better in CV results. 

", "rawMarkdown": "", "votes": null }, { "id": "65832", "postDate": "03/10/2015 01:27:17", "content": "

How do you guys get your CV LogLoss? I just got accuracies. About 99%

", "rawMarkdown": "", "votes": null }, { "id": "65839", "postDate": "03/10/2015 02:26:11", "content": "

[quote=Jiming Ye;65832]

\n

How do you guys get your CV LogLoss? I just got accuracies. About 99%

\n

[/quote]

\n

Hi, I use the code below, where y_true is Nx1 true label vector with value ranging from 0 to 8, and y_pred is Nx9 prediction arrays.

\n

import numpy as np

\n

from math import log, exp, sqrt,factorial
def multiclass_log_loss(y_true, y_pred, eps=1e-15):

\n

      predictions = np.clip(y_pred, eps, 1 - eps)

\n

      # normalize row sums to 1
      predictions /= predictions.sum(axis=1)[:, np.newaxis]

\n

      actual = np.zeros(y_pred.shape)
      n_samples = actual.shape[0]

      actual[np.arange(n_samples), y_true.astype(int)] = 1
      vectsum = np.sum(actual * np.log(predictions))
      loss = -1.0 / n_samples * vectsum
      return loss

\n

I steal it from the tutorial of the Science Bowl contest.

", "rawMarkdown": "", "votes": null }, { "id": "65840", "postDate": "03/10/2015 02:28:34", "content": "

Not sure if this is what you're asking... In Python:

\n

from sklearn.metrics import log_loss

", "rawMarkdown": "", "votes": null }, { "id": "65841", "postDate": "03/10/2015 02:28:51", "content": "

[quote=Bats & Robots;65815]

\n

I'm getting 0.02315174 in a 5-fold cv, but my LB score is ~0.109...i'm not sure why =p

\n

[/quote]

\n

you may want to check your test data. they can be flawed during downloading.

", "rawMarkdown": "", "votes": null }, { "id": "65842", "postDate": "03/10/2015 02:40:08", "content": "

[quote=rcarson;65839]

\n

[quote=Jiming Ye;65832]

\n

How do you guys get your CV LogLoss? I just got accuracies. About 99%

\n

[/quote]

\n

Hi, I use the code below, where y_true is Nx1 true label vector with value ranging from 0 to 8, and y_pred is Nx9 prediction arrays.

\n

import numpy as np

\n

from math import log, exp, sqrt,factorial
def multiclass_log_loss(y_true, y_pred, eps=1e-15):

\n

      predictions = np.clip(y_pred, eps, 1 - eps)

\n

      # normalize row sums to 1
      predictions /= predictions.sum(axis=1)[:, np.newaxis]

\n

      actual = np.zeros(y_pred.shape)
      n_samples = actual.shape[0]

      actual[np.arange(n_samples), y_true.astype(int)] = 1
      vectsum = np.sum(actual * np.log(predictions))
      loss = -1.0 / n_samples * vectsum
      return loss

\n

I steal it from the tutorial of the Science Bowl contest.

\n

[/quote] I did exactly the same thing!

", "rawMarkdown": "", "votes": null }, { "id": "65845", "postDate": "03/10/2015 02:43:18", "content": "

[quote=Little Boat;65842]

\n

I did exactly the same thing!

\n

[/quote]

\n

Well done! gogogo :D

", "rawMarkdown": "", "votes": null }, { "id": "65855", "postDate": "03/10/2015 03:50:20", "content": "

[quote=rcarson;65839]

\n

[quote=Jiming Ye;65832]

\n

How do you guys get your CV LogLoss? I just got accuracies. About 99%

\n

[/quote]

\n

Hi, I use the code below, where y_true is Nx1 true label vector with value ranging from 0 to 8, and y_pred is Nx9 prediction arrays.

\n

import numpy as np

\n

from math import log, exp, sqrt,factorial
def multiclass_log_loss(y_true, y_pred, eps=1e-15):

\n

      predictions = np.clip(y_pred, eps, 1 - eps)

\n

      # normalize row sums to 1
      predictions /= predictions.sum(axis=1)[:, np.newaxis]

\n

      actual = np.zeros(y_pred.shape)
      n_samples = actual.shape[0]

      actual[np.arange(n_samples), y_true.astype(int)] = 1
      vectsum = np.sum(actual * np.log(predictions))
      loss = -1.0 / n_samples * vectsum
      return loss

\n

I steal it from the tutorial of the Science Bowl contest.

\n

[/quote]

\n

Great! Thanks so much

", "rawMarkdown": "", "votes": null }, { "id": "65979", "postDate": "03/11/2015 20:28:46", "content": "

[quote=rcarson;65841]

\n

[quote=Bats & Robots;65815]

\n

I'm getting 0.02315174 in a 5-fold cv, but my LB score is ~0.109...i'm not sure why =p

\n

[/quote]

\n

you may want to check your test data. they can be flawed during downloading.

\n

[/quote]

\n\n

I did check the md5 hashes, they are ok. I added some features, then my new CV got 0.0196 but now i got 0.11 as LB. I'm using SVM...any hints?

\n

thanks!

", "rawMarkdown": "", "votes": null }, { "id": "65985", "postDate": "03/11/2015 21:13:37", "content": "

0.11 is actually low for any kind of messing up of data or id or label stuff I could think of. it seems more like overfitting to me now. Anyway maybe you could check whether the distribution of each class in submission is similar to the training data set. Or manually check if there are any weird thing in submission file. Ideally for a 0.0196 cv you should see that most files are predicted >0.99 probability to only one class. 

", "rawMarkdown": "", "votes": null }, { "id": "65988", "postDate": "03/11/2015 21:45:44", "content": "

[quote=rcarson;65985]

\n

0.11 is actually low for any kind of messing up of data or id or label stuff I could think of. it seems more like overfitting to me now. Anyway maybe you could check whether the distribution of each class in submission is similar to the training data set. Or manually check if there are any weird thing in submission file. Ideally for a 0.0196 cv you should see that most files are predicted >0.99 probability to only one class. 

\n

[/quote]

\n

I managed to get ~0.08 on LB after removing some weird normalization and changing models. Now i'll re-run the SVM without this normalization mess (since it will perform scaling by default). Also the submission file has a similar distribution as the train set. Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "66019", "postDate": "03/12/2015 08:16:40", "content": "

[quote=Bats & Robots;65979]

\n

I did check the md5 hashes, they are ok. I added some features, then my new CV got 0.0196 but now i got 0.11 as LB. I'm using SVM...any hints?

\n

thanks!

\n

[/quote]

\n\n

If it's not overfitting, it could be some mistake in the CV process. Take care to not use the validation data during training in any way.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 65778, "author_name": "jiweiliu", "author_url": "", "post_date": "03/09/2015 14:14:39", "content": "

my 4-fold CV is about 0.0007~0.001 higher than LB

", "votes": null, "replies": [] }, { "id": 65810, "author_name": "jieqchen", "author_url": "", "post_date": "03/09/2015 19:32:39", "content": "

@Stergios, 

\n

Mine (10-fold) CV score compared to LB is similar to your situation. 

", "votes": null, "replies": [] }, { "id": 65815, "author_name": "snowdog", "author_url": "", "post_date": "03/09/2015 21:28:04", "content": "

I'm getting 0.02315174 in a 5-fold cv, but my LB score is ~0.109...i'm not sure why =p

", "votes": null, "replies": [] }, { "id": 65817, "author_name": "xiaozhouwang", "author_url": "", "post_date": "03/09/2015 22:37:15", "content": "

Mine is also about 0.001 -- 0.002 better in CV results. 

", "votes": null, "replies": [] }, { "id": 65832, "author_name": "yejiming", "author_url": "", "post_date": "03/10/2015 01:27:17", "content": "

How do you guys get your CV LogLoss? I just got accuracies. About 99%

", "votes": null, "replies": [] }, { "id": 65839, "author_name": "jiweiliu", "author_url": "", "post_date": "03/10/2015 02:26:11", "content": "

[quote=Jiming Ye;65832]

\n

How do you guys get your CV LogLoss? I just got accuracies. About 99%

\n

[/quote]

\n

Hi, I use the code below, where y_true is Nx1 true label vector with value ranging from 0 to 8, and y_pred is Nx9 prediction arrays.

\n

import numpy as np

\n

from math import log, exp, sqrt,factorial
def multiclass_log_loss(y_true, y_pred, eps=1e-15):

\n

      predictions = np.clip(y_pred, eps, 1 - eps)

\n

      # normalize row sums to 1
      predictions /= predictions.sum(axis=1)[:, np.newaxis]

\n

      actual = np.zeros(y_pred.shape)
      n_samples = actual.shape[0]

      actual[np.arange(n_samples), y_true.astype(int)] = 1
      vectsum = np.sum(actual * np.log(predictions))
      loss = -1.0 / n_samples * vectsum
      return loss

\n

I steal it from the tutorial of the Science Bowl contest.

", "votes": null, "replies": [] }, { "id": 65840, "author_name": "jieqchen", "author_url": "", "post_date": "03/10/2015 02:28:34", "content": "

Not sure if this is what you're asking... In Python:

\n

from sklearn.metrics import log_loss

", "votes": null, "replies": [] }, { "id": 65841, "author_name": "jiweiliu", "author_url": "", "post_date": "03/10/2015 02:28:51", "content": "

[quote=Bats & Robots;65815]

\n

I'm getting 0.02315174 in a 5-fold cv, but my LB score is ~0.109...i'm not sure why =p

\n

[/quote]

\n

you may want to check your test data. they can be flawed during downloading.

", "votes": null, "replies": [] }, { "id": 65842, "author_name": "xiaozhouwang", "author_url": "", "post_date": "03/10/2015 02:40:08", "content": "

[quote=rcarson;65839]

\n

[quote=Jiming Ye;65832]

\n

How do you guys get your CV LogLoss? I just got accuracies. About 99%

\n

[/quote]

\n

Hi, I use the code below, where y_true is Nx1 true label vector with value ranging from 0 to 8, and y_pred is Nx9 prediction arrays.

\n

import numpy as np

\n

from math import log, exp, sqrt,factorial
def multiclass_log_loss(y_true, y_pred, eps=1e-15):

\n

      predictions = np.clip(y_pred, eps, 1 - eps)

\n

      # normalize row sums to 1
      predictions /= predictions.sum(axis=1)[:, np.newaxis]

\n

      actual = np.zeros(y_pred.shape)
      n_samples = actual.shape[0]

      actual[np.arange(n_samples), y_true.astype(int)] = 1
      vectsum = np.sum(actual * np.log(predictions))
      loss = -1.0 / n_samples * vectsum
      return loss

\n

I steal it from the tutorial of the Science Bowl contest.

\n

[/quote] I did exactly the same thing!

", "votes": null, "replies": [] }, { "id": 65845, "author_name": "jiweiliu", "author_url": "", "post_date": "03/10/2015 02:43:18", "content": "

[quote=Little Boat;65842]

\n

I did exactly the same thing!

\n

[/quote]

\n

Well done! gogogo :D

", "votes": null, "replies": [] }, { "id": 65855, "author_name": "yejiming", "author_url": "", "post_date": "03/10/2015 03:50:20", "content": "

[quote=rcarson;65839]

\n

[quote=Jiming Ye;65832]

\n

How do you guys get your CV LogLoss? I just got accuracies. About 99%

\n

[/quote]

\n

Hi, I use the code below, where y_true is Nx1 true label vector with value ranging from 0 to 8, and y_pred is Nx9 prediction arrays.

\n

import numpy as np

\n

from math import log, exp, sqrt,factorial
def multiclass_log_loss(y_true, y_pred, eps=1e-15):

\n

      predictions = np.clip(y_pred, eps, 1 - eps)

\n

      # normalize row sums to 1
      predictions /= predictions.sum(axis=1)[:, np.newaxis]

\n

      actual = np.zeros(y_pred.shape)
      n_samples = actual.shape[0]

      actual[np.arange(n_samples), y_true.astype(int)] = 1
      vectsum = np.sum(actual * np.log(predictions))
      loss = -1.0 / n_samples * vectsum
      return loss

\n

I steal it from the tutorial of the Science Bowl contest.

\n

[/quote]

\n

Great! Thanks so much

", "votes": null, "replies": [] }, { "id": 65979, "author_name": "snowdog", "author_url": "", "post_date": "03/11/2015 20:28:46", "content": "

[quote=rcarson;65841]

\n

[quote=Bats & Robots;65815]

\n

I'm getting 0.02315174 in a 5-fold cv, but my LB score is ~0.109...i'm not sure why =p

\n

[/quote]

\n

you may want to check your test data. they can be flawed during downloading.

\n

[/quote]

\n\n

I did check the md5 hashes, they are ok. I added some features, then my new CV got 0.0196 but now i got 0.11 as LB. I'm using SVM...any hints?

\n

thanks!

", "votes": null, "replies": [] }, { "id": 65985, "author_name": "jiweiliu", "author_url": "", "post_date": "03/11/2015 21:13:37", "content": "

0.11 is actually low for any kind of messing up of data or id or label stuff I could think of. it seems more like overfitting to me now. Anyway maybe you could check whether the distribution of each class in submission is similar to the training data set. Or manually check if there are any weird thing in submission file. Ideally for a 0.0196 cv you should see that most files are predicted >0.99 probability to only one class. 

", "votes": null, "replies": [] }, { "id": 65988, "author_name": "snowdog", "author_url": "", "post_date": "03/11/2015 21:45:44", "content": "

[quote=rcarson;65985]

\n

0.11 is actually low for any kind of messing up of data or id or label stuff I could think of. it seems more like overfitting to me now. Anyway maybe you could check whether the distribution of each class in submission is similar to the training data set. Or manually check if there are any weird thing in submission file. Ideally for a 0.0196 cv you should see that most files are predicted >0.99 probability to only one class. 

\n

[/quote]

\n

I managed to get ~0.08 on LB after removing some weird normalization and changing models. Now i'll re-run the SVM without this normalization mess (since it will perform scaling by default). Also the submission file has a similar distribution as the train set. Thanks!

", "votes": null, "replies": [] }, { "id": 66019, "author_name": "asterios", "author_url": "", "post_date": "03/12/2015 08:16:40", "content": "

[quote=Bats & Robots;65979]

\n

I did check the md5 hashes, they are ok. I added some features, then my new CV got 0.0196 but now i got 0.11 as LB. I'm using SVM...any hints?

\n

thanks!

\n

[/quote]

\n\n

If it's not overfitting, it could be some mistake in the CV process. Take care to not use the validation data during training in any way.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "65767": "", "65778": "", "65810": "", "65815": "", "65817": "", "65832": "", "65839": "", "65840": "", "65841": "", "65842": "", "65845": "", "65855": "", "65979": "", "65985": "", "65988": "", "66019": "" }, "source": "meta" }