{ "id": 344545, "title": "A reminder for those of you who have used that 0.8 notebook just now.", "url": "/competitions/amex-default-prediction/discussion/344545", "author_name": "", "post_date": "2022-08-15T15:08:19.481000", "votes": 49, "comment_count": 56, "views": 0, "content": "

The last 0.800 high scoring notebooks had a problem, it did not correlate well with my notebook, only 0.8+, less than 0.9. Most likely it distorted the dataset of the private list, if you submit with this file, you can get a high score in the public list, but in the private list you will lose points. What an evil idea……

", "messages": [ { "id": 1899880, "postDate": "2022-08-15T15:08:19.480Z", "content": "

The last 0.800 high scoring notebooks had a problem, it did not correlate well with my notebook, only 0.8+, less than 0.9. Most likely it distorted the dataset of the private list, if you submit with this file, you can get a high score in the public list, but in the private list you will lose points. What an evil idea……

", "rawMarkdown": "The last 0.800 high scoring notebooks had a problem, it did not correlate well with my notebook, only 0.8+, less than 0.9. Most likely it distorted the dataset of the private list, if you submit with this file, you can get a high score in the public list, but in the private list you will lose points. What an evil idea......", "votes": 48 }, { "id": 1899982, "postDate": "2022-08-15T16:26:56.873Z", "content": "

I think sharing potentially useful information for a short period and then deleting is a generally toxic pattern and unfair to the community, so I'm going to repost the lgb focal loss code that was shared here a bit ago. I have not checked the code, used it, or thought about it much, so have no take yet on its actual usefulness. Please evaluate for yourself!

\n
from scipy.misc import derivative\n\ndef focal_loss_lgb(y_pred, dtrain, alpha=0.03, gamma=2):\n    a,g = alpha, gamma\n    y_true = dtrain.label\n    def fl(x,t):\n        p = 1/(1+np.exp(-x))\n        return -( a*t + (1-a)*(1-t) ) * (( 1 - ( t*p + (1-t)*(1-p)) )**g) * ( t*np.log(p)+(1-t)*np.log(1-p) )\n    partial_fl = lambda x: fl(x, y_true)\n    grad = derivative(partial_fl, y_pred, n=1, dx=1e-6)\n    hess = derivative(partial_fl, y_pred, n=2, dx=1e-6)\n    return grad, hess\n\n\ndef focal_loss_lgb_eval_error(y_pred, dtrain, alpha=0.03, gamma=2):\n    a,g = alpha, gamma\n    y_true = dtrain.label\n    p = 1/(1+np.exp(-y_pred))\n    loss = -( a*y_true + (1-a)*(1-y_true) ) * (( 1 - ( y_true*p + (1-y_true)*(1-p)) )**g) * ( y_true*np.log(p)+(1-y_true)*np.log(1-p) )\n    return 'focal_loss', np.mean(loss), False \n
", "rawMarkdown": "I think sharing potentially useful information for a short period and then deleting is a generally toxic pattern and unfair to the community, so I'm going to repost the lgb focal loss code that was shared here a bit ago. I have not checked the code, used it, or thought about it much, so have no take yet on its actual usefulness. Please evaluate for yourself!\n\n```\nfrom scipy.misc import derivative\n\ndef focal_loss_lgb(y_pred, dtrain, alpha=0.03, gamma=2):\n a,g = alpha, gamma\n y_true = dtrain.label\n def fl(x,t):\n p = 1/(1+np.exp(-x))\n return -( a*t + (1-a)*(1-t) ) * (( 1 - ( t*p + (1-t)*(1-p)) )**g) * ( t*np.log(p)+(1-t)*np.log(1-p) )\n partial_fl = lambda x: fl(x, y_true)\n grad = derivative(partial_fl, y_pred, n=1, dx=1e-6)\n hess = derivative(partial_fl, y_pred, n=2, dx=1e-6)\n return grad, hess\n\n\ndef focal_loss_lgb_eval_error(y_pred, dtrain, alpha=0.03, gamma=2):\n a,g = alpha, gamma\n y_true = dtrain.label\n p = 1/(1+np.exp(-y_pred))\n loss = -( a*y_true + (1-a)*(1-y_true) ) * (( 1 - ( y_true*p + (1-y_true)*(1-p)) )**g) * ( y_true*np.log(p)+(1-y_true)*np.log(1-p) )\n return 'focal_loss', np.mean(loss), False \n```", "votes": 18, "replies": [ { "id": 1899987, "postDate": "2022-08-15T16:30:26.573Z", "rawMarkdown": "", "votes": -21 }, { "id": 1899992, "postDate": "2022-08-15T16:33:44.740Z", "content": "

And then delete it?

", "rawMarkdown": "And then delete it?", "votes": 1 }, { "id": 1899997, "postDate": "2022-08-15T16:35:15.373Z", "content": "

Nah I’ll leave it up this time seeing as it has generated so much engagement!

", "rawMarkdown": "Nah I’ll leave it up this time seeing as it has generated so much engagement! ", "votes": -12 }, { "id": 1900087, "postDate": "2022-08-15T17:32:41.447Z", "content": "

I tried focal loss a month ago and it wasn't better than logloss. May be my code was wrong. I will try with the snippet you posted here.

", "rawMarkdown": "I tried focal loss a month ago and it wasn't better than logloss. May be my code was wrong. I will try with the snippet you posted here." }, { "id": 1900111, "postDate": "2022-08-15T17:51:10.437Z", "content": "

If you want to improve logloss you may just calibrate your score during CV (see calibratedclassifiercv). It may reduce logloss twice or more, but have no impact on AUC, AMEX. This competition about ranking.

", "rawMarkdown": "If you want to improve logloss you may just calibrate your score during CV (see **calibratedclassifiercv**). It may reduce logloss twice or more, but have no impact on AUC, AMEX. This competition about ranking.", "votes": 7 } ] }, { "id": 1900055, "postDate": "2022-08-15T17:11:59.627Z", "content": "

The submission was published twice from @julianmukaj @elmer7465 and deleted.
\nSo the author may have more accounts to use for submission.
\nAnd this is against the rules of the competition.

\n

Generally it may have no impact on top solutions but not ethical for 99% of contributors.

", "rawMarkdown": "The submission was published twice from @julianmukaj @elmer7465 and deleted.\nSo the author may have more accounts to use for submission.\nAnd this is against the rules of the competition.\n\nGenerally it may have no impact on top solutions but not ethical for 99% of contributors.", "votes": 4, "replies": [ { "id": 1900059, "postDate": "2022-08-15T17:18:38.597Z", "content": "

Nope someone else forked it.. Keep your baseless accusations to yourself. I literally published it because I knew people like that user would be forking it and sharing among others who do nothing but fork them.
\nAs you can see by about 75 accounts that submitted it.

", "rawMarkdown": "Nope someone else forked it.. Keep your baseless accusations to yourself. I literally published it because I knew people like that user would be forking it and sharing among others who do nothing but fork them.\nAs you can see by about 75 accounts that submitted it.", "votes": -13 }, { "id": 1900769, "postDate": "2022-08-16T08:36:09.440Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1913112, "postDate": "2022-08-25T06:38:17.117Z", "content": "

So how did it turn out? What private score did that submission get?

", "rawMarkdown": "So how did it turn out? What private score did that submission get?", "votes": 3 }, { "id": 1901153, "postDate": "2022-08-16T13:40:01.463Z", "content": "

Didn't use public submissions, but it's a shame for community. I never see same thing before here.
\nKaggle authorities should carefully examine this case.

", "rawMarkdown": "Didn't use public submissions, but it's a shame for community. I never see same thing before here.\nKaggle authorities should carefully examine this case. \n", "votes": 4 }, { "id": 1899884, "postDate": "2022-08-15T15:12:53.400Z", "content": "

Feeling very shocked. The community was supposed to be a great platform for learning and communication, and the atmosphere was broken.

", "rawMarkdown": "Feeling very shocked. The community was supposed to be a great platform for learning and communication, and the atmosphere was broken.", "votes": 3, "replies": [ { "id": 1899908, "postDate": "2022-08-15T15:30:57.040Z", "rawMarkdown": "", "votes": -2, "isDeleted": true }, { "id": 1900104, "postDate": "2022-08-15T17:44:37.603Z", "content": "

\"\"

\n

what are you doing？

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11227911%2F5f3b30b43519341f9b830f9b63be2114%2F052B861E-A233-4E30-8C3D-743873583787.jpeg?generation=1660585364130325&alt=media)\n\nwhat are you doing？" }, { "id": 1900108, "postDate": "2022-08-15T17:45:51.290Z", "content": "

Reminding you it’s a competition? Why are you shocked? Are people not allowed to submit notebooks if they score higher than yours?

\n

My comment was a bit rude hence I deleted it, the point was you are overreacting.

", "rawMarkdown": "Reminding you it’s a competition? Why are you shocked? Are people not allowed to submit notebooks if they score higher than yours?\n\nMy comment was a bit rude hence I deleted it, the point was you are overreacting.", "votes": -13 }, { "id": 1900116, "postDate": "2022-08-15T17:56:47.280Z", "content": "

As a newbie, I'm shocked at how normal it is for contestants to be able to penalize others for such behavior. I also never said I wouldn't allow people with higher scores than me to submit, so I don't know where you got that from.

\n

Language rudely scolded and then deleted it, for me it's like sending me a private one on one scolding me in general. Calm down please.

", "rawMarkdown": "As a newbie, I'm shocked at how normal it is for contestants to be able to penalize others for such behavior. I also never said I wouldn't allow people with higher scores than me to submit, so I don't know where you got that from.\n\nLanguage rudely scolded and then deleted it, for me it's like sending me a private one on one scolding me in general. Calm down please.", "votes": 4 }, { "id": 1900121, "postDate": "2022-08-15T18:03:26.480Z", "content": "

I think everyone should calm down, there have been many competitions where people post high ranking notebooks towards the end, surprised I am the first.. Throws a bit of spice in the final leg… It's also within the rules.. You are quite high ranking for having joined in the last 10 days eh?

\n

Apologies again for the language, not adding anymore to this conversation now enough has been discussed. Focus on your own submissions and not dodgy public high ranking ones..

", "rawMarkdown": "I think **everyone** should calm down, there have been many competitions where people post high ranking notebooks towards the end, surprised I am the first.. Throws a bit of spice in the final leg... It's also within the rules.. You are quite high ranking for having joined in the last 10 days eh?\n\nApologies again for the language, not adding anymore to this conversation now enough has been discussed. Focus on your own submissions and not dodgy public high ranking ones..\n\n", "votes": -16 }, { "id": 1901189, "postDate": "2022-08-16T13:57:42.833Z", "content": "

@julianmukaj its not the first time you behave poorly in discussions, try to point fingers at others then ask them to calm down. Maybe you should take a clue from the downvotes.

", "rawMarkdown": "@julianmukaj its not the first time you behave poorly in discussions, try to point fingers at others then ask them to calm down. Maybe you should take a clue from the downvotes. ", "votes": 3 }, { "id": 1905682, "postDate": "2022-08-19T08:24:56.580Z", "rawMarkdown": "", "votes": -4 }, { "id": 1905933, "postDate": "2022-08-19T12:57:22.403Z", "content": "

I did not use that word. And you were also being rude to new users.

", "rawMarkdown": "I did not use that word. And you were also being rude to new users. " }, { "id": 1910848, "postDate": "2022-08-23T18:13:05.373Z", "content": "

If someone carefully observes the clues of the private data set from the data you publish, and restores your original data, does it mean that he will get a medal? According to my observation, in the 0.80 notebook you released, the number you have changed with are not too much, and it is possible to recover it.

", "rawMarkdown": "If someone carefully observes the clues of the private data set from the data you publish, and restores your original data, does it mean that he will get a medal? According to my observation, in the 0.80 notebook you released, the number you have changed with are not too much, and it is possible to recover it." } ] }, { "id": 1902999, "postDate": "2022-08-17T03:41:54.663Z", "content": "

clever trick to throw people off

", "rawMarkdown": "clever trick to throw people off", "votes": 1 }, { "id": 1900224, "postDate": "2022-08-15T19:54:19.940Z", "content": "

always read the file path! - ITSATRAP

", "rawMarkdown": "always read the file path! - ITSATRAP", "votes": 1, "replies": [ { "id": 1900227, "postDate": "2022-08-15T19:58:17.137Z", "rawMarkdown": "", "votes": -15 } ] }, { "id": 1900147, "postDate": "2022-08-15T18:33:05.410Z", "content": "

There goes the Public LB 👀 . Just saw this and we dropped 3-4 places now .. Trust your CV seems now 😄

", "rawMarkdown": "There goes the Public LB 👀 . Just saw this and we dropped 3-4 places now .. Trust your CV seems now 😄", "votes": 1, "replies": [ { "id": 1901720, "postDate": "2022-08-16T20:47:45.923Z", "content": "

Also though not right but these public shake-ups cause us to get down and improve our models more 😄 and not just sit on the current position. In our case prompted us to improve more some of our models a bit more .

", "rawMarkdown": "Also though not right but these public shake-ups cause us to get down and improve our models more 😄 and not just sit on the current position. In our case prompted us to improve more some of our models a bit more .", "votes": 1 }, { "id": 1905685, "postDate": "2022-08-19T08:26:23.253Z", "content": "
\n

In our case prompted us to improve more some of our models a bit more .

\n
\n

The only team with the right attitude it seems 😄 Good luck!

", "rawMarkdown": ">In our case prompted us to improve more some of our models a bit more . \n\nThe only team with the right attitude it seems 😄 Good luck!", "votes": -3 } ] }, { "id": 1900083, "postDate": "2022-08-15T17:30:55.737Z", "content": "

I lost almost 50 positions because of this notebook :(

", "rawMarkdown": "I lost almost 50 positions because of this notebook :(", "votes": 2 }, { "id": 1899907, "postDate": "2022-08-15T15:30:26.783Z", "content": "

Why would you expect it to correlate highly with your work when you have a score of .799 and also have no idea on the model or features used to get that .800 score?

", "rawMarkdown": "Why would you expect it to correlate highly with your work when you have a score of .799 and also have no idea on the model or features used to get that .800 score?", "votes": -13, "replies": [ { "id": 1899921, "postDate": "2022-08-15T15:40:38.177Z", "content": "

This link may give us something to think about
\nYou are the person who just shared your notebook with a score of 0.8 and I remember you well, I also have screenshots.
\nI gave my thoughts my suggestions, I have no right to interfere whether others will listen to me or not.
\nThank you!

", "rawMarkdown": "[This link may give us something to think about ](https://www.kaggle.com/competitions/amex-default-prediction/discussion/335698)\nYou are the person who just shared your notebook with a score of 0.8 and I remember you well, I also have screenshots.\nI gave my thoughts my suggestions, I have no right to interfere whether others will listen to me or not.\nThank you!", "votes": 5 }, { "id": 1899924, "postDate": "2022-08-15T15:43:23.387Z", "content": "
\n

I remember you well, I also have screenshots.

\n
\n

No need I will re-publish it for you

", "rawMarkdown": "> I remember you well, I also have screenshots.\n\nNo need I will re-publish it for you", "votes": -10 }, { "id": 1899925, "postDate": "2022-08-15T15:44:53.147Z", "content": "

No matter of correlation. In my opinion your action is not appropriate:

\n
    \n
  1. 0.80 score is too high for the last week of the competition and would provoke drastic lb changes for people that worked hard during the competition period.
    2. \n
  2. Your result is only csv file and not reproducible (to get oof predictions and verify results)
    3. \n
  3. You didn't disclose fe and model training and there is no learning from it
    4. \n
  4. Your comments have high grade of toxicity (just own opinion)
    5. \n
\n

I don't know why you did so - I can imagine that you did it for \"gags\", and it's just not a right thing.

", "rawMarkdown": "No matter of correlation. In my opinion your action is not appropriate:\n\n1. 0.80 score is too high for the last week of the competition and would provoke drastic lb changes for people that worked hard during the competition period.\n2. Your result is only csv file and not reproducible (to get oof predictions and verify results)\n3. You didn't disclose fe and model training and there is no learning from it\n4. Your comments have high grade of toxicity (just own opinion)\n\nI don't know why you did so - I can imagine that you did it for \"gags\", and it's just not a right thing.", "votes": 16 }, { "id": 1899927, "postDate": "2022-08-15T15:47:34.927Z", "content": "
\n

I don't know why you did so - I can imagine that you did it for \"gags\", and it's just not a right thing.

\n
\n

Not for gags, it was mainly published to penalize the users with fake accounts, plus its not the first time this has happened in Kaggle, since you are 3rd you have not much to worry about, are we all forgetting its a competition?

\n

Can't seem to re-publish it right now, maybe because ran out of submissions for today…

", "rawMarkdown": ">I don't know why you did so - I can imagine that you did it for \"gags\", and it's just not a right thing.\n\nNot for gags, it was mainly published to penalize the users with fake accounts, plus its not the first time this has happened in Kaggle, since you are 3rd you have not much to worry about, are we all forgetting its a competition?\n\nCan't seem to re-publish it right now, maybe because ran out of submissions for today...\n", "votes": -13 }, { "id": 1899959, "postDate": "2022-08-15T15:58:49.677Z", "content": "

Just curious how does it help penalize the users with fake accounts?

", "rawMarkdown": "Just curious how does it help penalize the users with fake accounts?", "votes": 3 }, { "id": 1899967, "postDate": "2022-08-15T16:07:16.130Z", "content": "

Every competition there’s these newly made accounts that are either just forking public notebooks or being used to circumvent the daily submission limits, most of them just seem to be trying a hill climb approach or whatever so we know how it will play out in private set for those that will use this.

", "rawMarkdown": "Every competition there’s these newly made accounts that are either just forking public notebooks or being used to circumvent the daily submission limits, most of them just seem to be trying a hill climb approach or whatever so we know how it will play out in private set for those that will use this.", "votes": -8 }, { "id": 1900008, "postDate": "2022-08-15T16:40:59.940Z", "content": "
\n

are we all forgetting its a competition?

\n
\n

Sure it is a competition. But most people compete without trying to sabotage others. Why do you take it on yourself to penalize other participants?

", "rawMarkdown": "> are we all forgetting its a competition?\n\nSure it is a competition. But most people compete without trying to sabotage others. Why do you take it on yourself to `penalize` other participants?", "votes": 1 }, { "id": 1900013, "postDate": "2022-08-15T16:42:07.073Z", "content": "

It only penalises those who only fork public notebooks, if you stick with your own work you won’t get any surprises..

", "rawMarkdown": "It only penalises those who only fork public notebooks, if you stick with your own work you won’t get any surprises..", "votes": -10 } ] }, { "id": 1906766, "postDate": "2022-08-20T07:21:03.907Z", "content": "

hi.

", "rawMarkdown": "hi. ", "votes": -1 }, { "id": 1905500, "postDate": "2022-08-19T05:25:36.490Z", "content": "

Helloo Everyone

", "rawMarkdown": "Helloo Everyone\n", "votes": -3 }, { "id": 1901334, "postDate": "2022-08-16T15:29:22.710Z", "content": "

That is amazing!

", "rawMarkdown": "That is amazing!", "votes": -4 }, { "id": 1900113, "postDate": "2022-08-15T17:54:18.620Z", "content": "

https://www.kaggle.com/code/julianmukaj/toxic-shakeup

\n

Since you can't downvote notebooks, I will give you all a chance to downvote me on this comment instead!

\n

Have fun and see you at the end. 9 days left, plenty of time to compete, not so much time to whine about public lb becoming useless.

", "rawMarkdown": "https://www.kaggle.com/code/julianmukaj/toxic-shakeup\n\nSince you can't downvote notebooks, I will give you all a chance to downvote me on this comment instead! \n\nHave fun and see you at the end. 9 days left, plenty of time to compete, not so much time to whine about public lb becoming useless.", "votes": -10, "replies": [ { "id": 1900137, "postDate": "2022-08-15T18:20:06.703Z", "content": "

In this case, you use Focal loss instead of amex loss function?
\nLike:
\n fobj = focal_loss_lgb,
\n feval = focal_loss_lgb_eval_error,

\n

Why focal loss works well in this dataset?
\ni read the original paper (https://arxiv.org/pdf/1708.02002.pdf) years ago and I thought focal loss worked better with unbalanced image datasets.

", "rawMarkdown": "In this case, you use Focal loss instead of amex loss function?\nLike:\n \tfobj = focal_loss_lgb,\n\t feval = focal_loss_lgb_eval_error,\n\nWhy focal loss works well in this dataset? \ni read the original paper (https://arxiv.org/pdf/1708.02002.pdf) years ago and I thought focal loss worked better with unbalanced image datasets.", "votes": 2 }, { "id": 1900146, "postDate": "2022-08-15T18:30:39.213Z", "content": "

To be clear, I am not saying using this objective function will get you a score of 0.800.. I also never said its better than logloss but someone complained I shared a notebook with no extra learning, so I added that because nobody had shared code for trying it out.

\n

Focal loss in theory should down weight on easy examples in the total loss, and focus on the harder ones by making their contribution to the loss greater. So it would be better at picking up the harder to classify defaults in the dataset. You can use that info in your ensemble however it may work for you or not.

", "rawMarkdown": "To be clear, I am not saying using this objective function will get you a score of 0.800.. I also never said its better than logloss but someone complained I shared a notebook with no extra learning, so I added that because nobody had shared code for trying it out. \n\nFocal loss in theory should down weight on easy examples in the total loss, and focus on the harder ones by making their contribution to the loss greater. So it would be better at picking up the harder to classify defaults in the dataset. You can use that info in your ensemble however it may work for you or not.", "votes": -6 }, { "id": 1900155, "postDate": "2022-08-15T18:39:47.503Z", "content": "

The fact is that it's because you're distorting private leaderboard data to punish people that people are complaining, if it was NO extra learning notebooks people wouldn't say anything.

", "rawMarkdown": "The fact is that it's because you're distorting private leaderboard data to punish people that people are complaining, if it was NO extra learning notebooks people wouldn't say anything.", "votes": 2 }, { "id": 1900166, "postDate": "2022-08-15T18:46:33.390Z", "content": "

Then the people who actually tried and got 0.799 on their CV will more likely get a medal if the ones who just fork are at the bottom then won't they?

", "rawMarkdown": "Then the people who actually tried and got 0.799 on their CV will more likely get a medal if the ones who just fork are at the bottom then won't they?", "votes": -7 } ] }, { "id": 1922288, "postDate": "2022-09-01T11:45:29.063Z", "content": "

Hi @chal1ce May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants

", "rawMarkdown": "Hi @chal1ce May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants" }, { "id": 1906858, "postDate": "2022-08-20T09:06:36.003Z", "content": "

I think it'll be a big deal

", "rawMarkdown": "I think it'll be a big deal " }, { "id": 1906062, "postDate": "2022-08-19T14:35:09.090Z", "content": "

oh my god thats bad

", "rawMarkdown": "oh my god thats bad" }, { "id": 1904504, "postDate": "2022-08-18T08:26:45.453Z", "content": "

this is a really friendly reminder.

", "rawMarkdown": "this is a really friendly reminder." }, { "id": 1903155, "postDate": "2022-08-17T06:42:56.820Z", "content": "

@chal1ce I'm not defending nor attacking the person who published the high scoring notebook but

\n
\n

Most likely it distorted the dataset of the private list

\n
\n

By this do you mean a public LB overfit (which I suspect quite a few public notebooks are)? Or do you mean some kind of manually overwritten probabilities to improve the public LB score?

", "rawMarkdown": "@chal1ce I'm not defending nor attacking the person who published the high scoring notebook but\n\n> Most likely it distorted the dataset of the private list\n\nBy this do you mean a public LB overfit (which I suspect quite a few public notebooks are)? Or do you mean some kind of manually overwritten probabilities to improve the public LB score?", "replies": [ { "id": 1903284, "postDate": "2022-08-17T09:11:10.740Z", "content": "

Hi Chi.

\n
\n

This leaderboard is calculated with approximately 51% of the test data. The final results will be based on the other 49%, so the final standings may be different.>
\n There was a mention of this in the introduction to the competition.

\n
\n

This link may let you know more

\n

Because a file contains data for both public lb and private lb, the dataset we test on public is independent of the dataset tested on private. This means that he can modify the private data as he pleases, and although the score from the public lb is high, you will know how low the score on the private lb is when the game is over.

", "rawMarkdown": "Hi Chi.\n> This leaderboard is calculated with approximately 51% of the test data. The final results will be based on the other 49%, so the final standings may be different.> \nThere was a mention of this in the introduction to the competition.\n\n[This link may let you know more](https://www.kaggle.com/competitions/amex-default-prediction/discussion/335698)\n\nBecause a file contains data for both public lb and private lb, the dataset we test on public is independent of the dataset tested on private. This means that he can modify the private data as he pleases, and although the score from the public lb is high, you will know how low the score on the private lb is when the game is over.", "votes": 1 } ] }, { "id": 1900829, "postDate": "2022-08-16T09:31:07.200Z", "content": "

just delete it!

", "rawMarkdown": "just delete it!" }, { "id": 1910781, "postDate": "2022-08-23T17:07:32.370Z", "content": "

Thank you for your kindness.

", "rawMarkdown": "Thank you for your kindness." }, { "id": 1910776, "postDate": "2022-08-23T17:05:43.687Z", "content": "

A kind reminder, thanks

", "rawMarkdown": "A kind reminder, thanks" }, { "id": 1909794, "postDate": "2022-08-22T23:53:11.213Z", "content": "

Thanks a Lot !!

", "rawMarkdown": "Thanks a Lot !!" }, { "id": 1906020, "postDate": "2022-08-19T14:10:24.640Z", "content": "

thanks for your nice information

", "rawMarkdown": "thanks for your nice information" } ], "comments": [ { "id": 1899982, "author_name": "Joe Eddy", "author_url": "", "post_date": "2022-08-15T16:26:56.873000", "content": "

I think sharing potentially useful information for a short period and then deleting is a generally toxic pattern and unfair to the community, so I'm going to repost the lgb focal loss code that was shared here a bit ago. I have not checked the code, used it, or thought about it much, so have no take yet on its actual usefulness. Please evaluate for yourself!

\n
from scipy.misc import derivative\n\ndef focal_loss_lgb(y_pred, dtrain, alpha=0.03, gamma=2):\n    a,g = alpha, gamma\n    y_true = dtrain.label\n    def fl(x,t):\n        p = 1/(1+np.exp(-x))\n        return -( a*t + (1-a)*(1-t) ) * (( 1 - ( t*p + (1-t)*(1-p)) )**g) * ( t*np.log(p)+(1-t)*np.log(1-p) )\n    partial_fl = lambda x: fl(x, y_true)\n    grad = derivative(partial_fl, y_pred, n=1, dx=1e-6)\n    hess = derivative(partial_fl, y_pred, n=2, dx=1e-6)\n    return grad, hess\n\n\ndef focal_loss_lgb_eval_error(y_pred, dtrain, alpha=0.03, gamma=2):\n    a,g = alpha, gamma\n    y_true = dtrain.label\n    p = 1/(1+np.exp(-y_pred))\n    loss = -( a*y_true + (1-a)*(1-y_true) ) * (( 1 - ( y_true*p + (1-y_true)*(1-p)) )**g) * ( y_true*np.log(p)+(1-y_true)*np.log(1-p) )\n    return 'focal_loss', np.mean(loss), False \n
", "votes": 18, "replies": [ { "id": 1899987, "author_name": "", "author_url": "", "post_date": "2022-08-15T16:30:26.573000", "content": "", "votes": -21, "replies": [] }, { "id": 1899992, "author_name": "Joe Eddy", "author_url": "", "post_date": "2022-08-15T16:33:44.740000", "content": "

And then delete it?

", "votes": 1, "replies": [] }, { "id": 1899997, "author_name": "JM", "author_url": "", "post_date": "2022-08-15T16:35:15.373000", "content": "

Nah I’ll leave it up this time seeing as it has generated so much engagement!

", "votes": -12, "replies": [] }, { "id": 1900087, "author_name": "nymfree", "author_url": "", "post_date": "2022-08-15T17:32:41.447000", "content": "

I tried focal loss a month ago and it wasn't better than logloss. May be my code was wrong. I will try with the snippet you posted here.

", "votes": 0, "replies": [] }, { "id": 1900111, "author_name": "Vladislav Bogorod", "author_url": "", "post_date": "2022-08-15T17:51:10.437000", "content": "

If you want to improve logloss you may just calibrate your score during CV (see calibratedclassifiercv). It may reduce logloss twice or more, but have no impact on AUC, AMEX. This competition about ranking.

", "votes": 7, "replies": [] } ] }, { "id": 1900055, "author_name": "Vladislav Bogorod", "author_url": "", "post_date": "2022-08-15T17:11:59.627000", "content": "

The submission was published twice from @julianmukaj @elmer7465 and deleted.
\nSo the author may have more accounts to use for submission.
\nAnd this is against the rules of the competition.

\n

Generally it may have no impact on top solutions but not ethical for 99% of contributors.

", "votes": 4, "replies": [ { "id": 1900059, "author_name": "JM", "author_url": "", "post_date": "2022-08-15T17:18:38.597000", "content": "

Nope someone else forked it.. Keep your baseless accusations to yourself. I literally published it because I knew people like that user would be forking it and sharing among others who do nothing but fork them.
\nAs you can see by about 75 accounts that submitted it.

", "votes": -13, "replies": [] }, { "id": 1900769, "author_name": "", "author_url": "", "post_date": "2022-08-16T08:36:09.440000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1913112, "author_name": "Arigion", "author_url": "", "post_date": "2022-08-25T06:38:17.117000", "content": "

So how did it turn out? What private score did that submission get?

", "votes": 3, "replies": [] }, { "id": 1901153, "author_name": "Sasha Turutin", "author_url": "", "post_date": "2022-08-16T13:40:01.463000", "content": "

Didn't use public submissions, but it's a shame for community. I never see same thing before here.
\nKaggle authorities should carefully examine this case.

", "votes": 4, "replies": [] }, { "id": 1899884, "author_name": "", "author_url": "", "post_date": "2022-08-15T15:12:53.400000", "content": "

Feeling very shocked. The community was supposed to be a great platform for learning and communication, and the atmosphere was broken.

", "votes": 3, "replies": [ { "id": 1899908, "author_name": "", "author_url": "", "post_date": "2022-08-15T15:30:57.040000", "content": "", "votes": -2, "replies": [] }, { "id": 1900104, "author_name": "", "author_url": "", "post_date": "2022-08-15T17:44:37.603000", "content": "

\"\"

\n

what are you doing？

", "votes": 0, "replies": [] }, { "id": 1900108, "author_name": "JM", "author_url": "", "post_date": "2022-08-15T17:45:51.290000", "content": "

Reminding you it’s a competition? Why are you shocked? Are people not allowed to submit notebooks if they score higher than yours?

\n

My comment was a bit rude hence I deleted it, the point was you are overreacting.

", "votes": -13, "replies": [] }, { "id": 1900116, "author_name": "", "author_url": "", "post_date": "2022-08-15T17:56:47.280000", "content": "

As a newbie, I'm shocked at how normal it is for contestants to be able to penalize others for such behavior. I also never said I wouldn't allow people with higher scores than me to submit, so I don't know where you got that from.

\n

Language rudely scolded and then deleted it, for me it's like sending me a private one on one scolding me in general. Calm down please.

", "votes": 4, "replies": [] }, { "id": 1900121, "author_name": "JM", "author_url": "", "post_date": "2022-08-15T18:03:26.480000", "content": "

I think everyone should calm down, there have been many competitions where people post high ranking notebooks towards the end, surprised I am the first.. Throws a bit of spice in the final leg… It's also within the rules.. You are quite high ranking for having joined in the last 10 days eh?

\n

Apologies again for the language, not adding anymore to this conversation now enough has been discussed. Focus on your own submissions and not dodgy public high ranking ones..

", "votes": -16, "replies": [] }, { "id": 1901189, "author_name": "Lucas Morin", "author_url": "", "post_date": "2022-08-16T13:57:42.833000", "content": "

@julianmukaj its not the first time you behave poorly in discussions, try to point fingers at others then ask them to calm down. Maybe you should take a clue from the downvotes.

", "votes": 3, "replies": [] }, { "id": 1905682, "author_name": "", "author_url": "", "post_date": "2022-08-19T08:24:56.580000", "content": "", "votes": -4, "replies": [] }, { "id": 1905933, "author_name": "Lucas Morin", "author_url": "", "post_date": "2022-08-19T12:57:22.403000", "content": "

I did not use that word. And you were also being rude to new users.

", "votes": 0, "replies": [] }, { "id": 1910848, "author_name": "shize facai", "author_url": "", "post_date": "2022-08-23T18:13:05.373000", "content": "

If someone carefully observes the clues of the private data set from the data you publish, and restores your original data, does it mean that he will get a medal? According to my observation, in the 0.80 notebook you released, the number you have changed with are not too much, and it is possible to recover it.

", "votes": 0, "replies": [] } ] }, { "id": 1902999, "author_name": "Kyle Montag", "author_url": "", "post_date": "2022-08-17T03:41:54.663000", "content": "

clever trick to throw people off

", "votes": 1, "replies": [] }, { "id": 1900224, "author_name": "Jake", "author_url": "", "post_date": "2022-08-15T19:54:19.940000", "content": "

always read the file path! - ITSATRAP

", "votes": 1, "replies": [ { "id": 1900227, "author_name": "", "author_url": "", "post_date": "2022-08-15T19:58:17.137000", "content": "", "votes": -15, "replies": [] } ] }, { "id": 1900147, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2022-08-15T18:33:05.410000", "content": "

There goes the Public LB 👀 . Just saw this and we dropped 3-4 places now .. Trust your CV seems now 😄

", "votes": 1, "replies": [ { "id": 1901720, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2022-08-16T20:47:45.923000", "content": "

Also though not right but these public shake-ups cause us to get down and improve our models more 😄 and not just sit on the current position. In our case prompted us to improve more some of our models a bit more .

", "votes": 1, "replies": [] }, { "id": 1905685, "author_name": "JM", "author_url": "", "post_date": "2022-08-19T08:26:23.253000", "content": "
\n

In our case prompted us to improve more some of our models a bit more .

\n
\n

The only team with the right attitude it seems 😄 Good luck!

", "votes": -3, "replies": [] } ] }, { "id": 1900083, "author_name": "Fernando Melo", "author_url": "", "post_date": "2022-08-15T17:30:55.737000", "content": "

I lost almost 50 positions because of this notebook :(

", "votes": 2, "replies": [] }, { "id": 1899907, "author_name": "JM", "author_url": "", "post_date": "2022-08-15T15:30:26.783000", "content": "

Why would you expect it to correlate highly with your work when you have a score of .799 and also have no idea on the model or features used to get that .800 score?

", "votes": -13, "replies": [ { "id": 1899921, "author_name": "", "author_url": "", "post_date": "2022-08-15T15:40:38.177000", "content": "

This link may give us something to think about
\nYou are the person who just shared your notebook with a score of 0.8 and I remember you well, I also have screenshots.
\nI gave my thoughts my suggestions, I have no right to interfere whether others will listen to me or not.
\nThank you!

", "votes": 5, "replies": [] }, { "id": 1899924, "author_name": "JM", "author_url": "", "post_date": "2022-08-15T15:43:23.387000", "content": "
\n

I remember you well, I also have screenshots.

\n
\n

No need I will re-publish it for you

", "votes": -10, "replies": [] }, { "id": 1899925, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2022-08-15T15:44:53.147000", "content": "

No matter of correlation. In my opinion your action is not appropriate:

\n
    \n
  1. 0.80 score is too high for the last week of the competition and would provoke drastic lb changes for people that worked hard during the competition period.
    2. \n
  2. Your result is only csv file and not reproducible (to get oof predictions and verify results)
    3. \n
  3. You didn't disclose fe and model training and there is no learning from it
    4. \n
  4. Your comments have high grade of toxicity (just own opinion)
    5. \n
\n

I don't know why you did so - I can imagine that you did it for \"gags\", and it's just not a right thing.

", "votes": 16, "replies": [] }, { "id": 1899927, "author_name": "JM", "author_url": "", "post_date": "2022-08-15T15:47:34.927000", "content": "
\n

I don't know why you did so - I can imagine that you did it for \"gags\", and it's just not a right thing.

\n
\n

Not for gags, it was mainly published to penalize the users with fake accounts, plus its not the first time this has happened in Kaggle, since you are 3rd you have not much to worry about, are we all forgetting its a competition?

\n

Can't seem to re-publish it right now, maybe because ran out of submissions for today…

", "votes": -13, "replies": [] }, { "id": 1899959, "author_name": "Tonghui Li", "author_url": "", "post_date": "2022-08-15T15:58:49.677000", "content": "

Just curious how does it help penalize the users with fake accounts?

", "votes": 3, "replies": [] }, { "id": 1899967, "author_name": "JM", "author_url": "", "post_date": "2022-08-15T16:07:16.130000", "content": "

Every competition there’s these newly made accounts that are either just forking public notebooks or being used to circumvent the daily submission limits, most of them just seem to be trying a hill climb approach or whatever so we know how it will play out in private set for those that will use this.

", "votes": -8, "replies": [] }, { "id": 1900008, "author_name": "Tonghui Li", "author_url": "", "post_date": "2022-08-15T16:40:59.940000", "content": "
\n

are we all forgetting its a competition?

\n
\n

Sure it is a competition. But most people compete without trying to sabotage others. Why do you take it on yourself to penalize other participants?

", "votes": 1, "replies": [] }, { "id": 1900013, "author_name": "JM", "author_url": "", "post_date": "2022-08-15T16:42:07.073000", "content": "

It only penalises those who only fork public notebooks, if you stick with your own work you won’t get any surprises..

", "votes": -10, "replies": [] } ] }, { "id": 1906766, "author_name": "Prateek Khandelwal", "author_url": "", "post_date": "2022-08-20T07:21:03.907000", "content": "

hi.

", "votes": -1, "replies": [] }, { "id": 1905500, "author_name": "M Karthik Kumar", "author_url": "", "post_date": "2022-08-19T05:25:36.490000", "content": "

Helloo Everyone

", "votes": -3, "replies": [] }, { "id": 1901334, "author_name": "金穗辉辉", "author_url": "", "post_date": "2022-08-16T15:29:22.710000", "content": "

That is amazing!

", "votes": -4, "replies": [] }, { "id": 1900113, "author_name": "JM", "author_url": "", "post_date": "2022-08-15T17:54:18.620000", "content": "

https://www.kaggle.com/code/julianmukaj/toxic-shakeup

\n

Since you can't downvote notebooks, I will give you all a chance to downvote me on this comment instead!

\n

Have fun and see you at the end. 9 days left, plenty of time to compete, not so much time to whine about public lb becoming useless.

", "votes": -10, "replies": [ { "id": 1900137, "author_name": "Fernando Melo", "author_url": "", "post_date": "2022-08-15T18:20:06.703000", "content": "

In this case, you use Focal loss instead of amex loss function?
\nLike:
\n fobj = focal_loss_lgb,
\n feval = focal_loss_lgb_eval_error,

\n

Why focal loss works well in this dataset?
\ni read the original paper (https://arxiv.org/pdf/1708.02002.pdf) years ago and I thought focal loss worked better with unbalanced image datasets.

", "votes": 2, "replies": [] }, { "id": 1900146, "author_name": "JM", "author_url": "", "post_date": "2022-08-15T18:30:39.213000", "content": "

To be clear, I am not saying using this objective function will get you a score of 0.800.. I also never said its better than logloss but someone complained I shared a notebook with no extra learning, so I added that because nobody had shared code for trying it out.

\n

Focal loss in theory should down weight on easy examples in the total loss, and focus on the harder ones by making their contribution to the loss greater. So it would be better at picking up the harder to classify defaults in the dataset. You can use that info in your ensemble however it may work for you or not.

", "votes": -6, "replies": [] }, { "id": 1900155, "author_name": "", "author_url": "", "post_date": "2022-08-15T18:39:47.503000", "content": "

The fact is that it's because you're distorting private leaderboard data to punish people that people are complaining, if it was NO extra learning notebooks people wouldn't say anything.

", "votes": 2, "replies": [] }, { "id": 1900166, "author_name": "JM", "author_url": "", "post_date": "2022-08-15T18:46:33.390000", "content": "

Then the people who actually tried and got 0.799 on their CV will more likely get a medal if the ones who just fork are at the bottom then won't they?

", "votes": -7, "replies": [] } ] }, { "id": 1922288, "author_name": "Yang Liu", "author_url": "", "post_date": "2022-09-01T11:45:29.063000", "content": "

Hi @chal1ce May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants

", "votes": 0, "replies": [] }, { "id": 1906858, "author_name": "sohaib soussi", "author_url": "", "post_date": "2022-08-20T09:06:36.003000", "content": "

I think it'll be a big deal

", "votes": 0, "replies": [] }, { "id": 1906062, "author_name": "Devanshu12122", "author_url": "", "post_date": "2022-08-19T14:35:09.090000", "content": "

oh my god thats bad

", "votes": 0, "replies": [] }, { "id": 1904504, "author_name": "IASD", "author_url": "", "post_date": "2022-08-18T08:26:45.453000", "content": "

this is a really friendly reminder.

", "votes": 0, "replies": [] }, { "id": 1903155, "author_name": "goto_conversion", "author_url": "", "post_date": "2022-08-17T06:42:56.820000", "content": "

@chal1ce I'm not defending nor attacking the person who published the high scoring notebook but

\n
\n

Most likely it distorted the dataset of the private list

\n
\n

By this do you mean a public LB overfit (which I suspect quite a few public notebooks are)? Or do you mean some kind of manually overwritten probabilities to improve the public LB score?

", "votes": 0, "replies": [ { "id": 1903284, "author_name": "", "author_url": "", "post_date": "2022-08-17T09:11:10.740000", "content": "

Hi Chi.

\n
\n

This leaderboard is calculated with approximately 51% of the test data. The final results will be based on the other 49%, so the final standings may be different.>
\n There was a mention of this in the introduction to the competition.

\n
\n

This link may let you know more

\n

Because a file contains data for both public lb and private lb, the dataset we test on public is independent of the dataset tested on private. This means that he can modify the private data as he pleases, and although the score from the public lb is high, you will know how low the score on the private lb is when the game is over.

", "votes": 1, "replies": [] } ] }, { "id": 1900829, "author_name": "PARKSAMJONG", "author_url": "", "post_date": "2022-08-16T09:31:07.200000", "content": "

just delete it!

", "votes": 0, "replies": [] }, { "id": 1910781, "author_name": "Bella Pisani", "author_url": "", "post_date": "2022-08-23T17:07:32.370000", "content": "

Thank you for your kindness.

", "votes": 0, "replies": [] }, { "id": 1910776, "author_name": "Qianye", "author_url": "", "post_date": "2022-08-23T17:05:43.687000", "content": "

A kind reminder, thanks

", "votes": 0, "replies": [] }, { "id": 1909794, "author_name": "RuiqiYY", "author_url": "", "post_date": "2022-08-22T23:53:11.213000", "content": "

Thanks a Lot !!

", "votes": 0, "replies": [] }, { "id": 1906020, "author_name": "gzw991029", "author_url": "", "post_date": "2022-08-19T14:10:24.640000", "content": "

thanks for your nice information

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1899880": "The last 0.800 high scoring notebooks had a problem, it did not correlate well with my notebook, only 0.8+, less than 0.9. Most likely it distorted the dataset of the private list, if you submit with this file, you can get a high score in the public list, but in the private list you will lose points. What an evil idea......", "1899982": "I think sharing potentially useful information for a short period and then deleting is a generally toxic pattern and unfair to the community, so I'm going to repost the lgb focal loss code that was shared here a bit ago. I have not checked the code, used it, or thought about it much, so have no take yet on its actual usefulness. Please evaluate for yourself!\n\n```\nfrom scipy.misc import derivative\n\ndef focal_loss_lgb(y_pred, dtrain, alpha=0.03, gamma=2):\n a,g = alpha, gamma\n y_true = dtrain.label\n def fl(x,t):\n p = 1/(1+np.exp(-x))\n return -( a*t + (1-a)*(1-t) ) * (( 1 - ( t*p + (1-t)*(1-p)) )**g) * ( t*np.log(p)+(1-t)*np.log(1-p) )\n partial_fl = lambda x: fl(x, y_true)\n grad = derivative(partial_fl, y_pred, n=1, dx=1e-6)\n hess = derivative(partial_fl, y_pred, n=2, dx=1e-6)\n return grad, hess\n\n\ndef focal_loss_lgb_eval_error(y_pred, dtrain, alpha=0.03, gamma=2):\n a,g = alpha, gamma\n y_true = dtrain.label\n p = 1/(1+np.exp(-y_pred))\n loss = -( a*y_true + (1-a)*(1-y_true) ) * (( 1 - ( y_true*p + (1-y_true)*(1-p)) )**g) * ( y_true*np.log(p)+(1-y_true)*np.log(1-p) )\n return 'focal_loss', np.mean(loss), False \n```", "1900055": "The submission was published twice from @julianmukaj @elmer7465 and deleted.\nSo the author may have more accounts to use for submission.\nAnd this is against the rules of the competition.\n\nGenerally it may have no impact on top solutions but not ethical for 99% of contributors.", "1913112": "So how did it turn out? What private score did that submission get?", "1901153": "Didn't use public submissions, but it's a shame for community. I never see same thing before here.\nKaggle authorities should carefully examine this case. \n", "1899884": "Feeling very shocked. The community was supposed to be a great platform for learning and communication, and the atmosphere was broken.", "1902999": "clever trick to throw people off", "1900224": "always read the file path! - ITSATRAP", "1900147": "There goes the Public LB 👀 . Just saw this and we dropped 3-4 places now .. Trust your CV seems now 😄", "1900083": "I lost almost 50 positions because of this notebook :(", "1899907": "Why would you expect it to correlate highly with your work when you have a score of .799 and also have no idea on the model or features used to get that .800 score?", "1906766": "hi. ", "1905500": "Helloo Everyone\n", "1901334": "That is amazing!", "1900113": "https://www.kaggle.com/code/julianmukaj/toxic-shakeup\n\nSince you can't downvote notebooks, I will give you all a chance to downvote me on this comment instead! \n\nHave fun and see you at the end. 9 days left, plenty of time to compete, not so much time to whine about public lb becoming useless.", "1922288": "Hi @chal1ce May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants", "1906858": "I think it'll be a big deal ", "1906062": "oh my god thats bad", "1904504": "this is a really friendly reminder.", "1903155": "@chal1ce I'm not defending nor attacking the person who published the high scoring notebook but\n\n> Most likely it distorted the dataset of the private list\n\nBy this do you mean a public LB overfit (which I suspect quite a few public notebooks are)? Or do you mean some kind of manually overwritten probabilities to improve the public LB score?", "1900829": "just delete it!", "1910781": "Thank you for your kindness.", "1910776": "A kind reminder, thanks", "1909794": "Thanks a Lot !!", "1906020": "thanks for your nice information" } }