Hi all!\nFirst congrats to all competitors and especially solo gold winners. They just did an amazing job. I especially thank to @ogrellier for his great kernels. First time I’ve used chunk method and multi-threading.
\n\nOur final submission was a blend of our single models and I had a single LGBM model with a CV of 0.493 and LB of 0.915. For me, most important boost came from using ratio of different passband features and using different sample weights instead of direct inverse ratio of class counts. Also using top 55 features as a feature selection method, gave another important boost.
\n\nMy teammates will explain details of models and features in another thread. In this thread I want to mention about postprocessing trick I’ve used and till now did not see in another thread. First I validated postprocessing on my OOF predictions and it also improved my LB score. After I was invited by the team, same process improved their models' both CV and LB.
\n\nOf course, as the model itself improves, gain from postprocess decreases. But, I think it will improve most of your models as well.
\n\nBefore this method, my best model had a CV of 0.52 and LB of 0.968 (Private: 0.99101). With this method CV moved to 0.493 and LB to 0.915 (Private: 0.93077) :)
\n\nI think what lies behind this trick to work is the competition metric. For each row, the correct class’s probability is multiplied with 1 and the rest with zero. That is, it does not matter how do you distribute probabilities among wrong classes. So with this operation, you can compensate the lack of your model by simply changing class preds with weights. For example, my model was giving higher probability for class 42 and class 90 most of the time and lowering their predictions always improved my CV.
\n\nHere is the code for your oof predictions. You can give a try:
\n\ndef change(preds):\n oof_df = pd.DataFrame(preds).copy()\n oof_df[3] = oof_df[3]* 1\n oof_df[11] = oof_df[11]* 1\n oof_df[0] = oof_df[0]* 1\n oof_df[1] = oof_df[1]* 1\n oof_df[2] = oof_df[2]* 1\n oof_df[4] = oof_df[4]* 1\n oof_df[5] = oof_df[5]* 1\n oof_df[6] = oof_df[6]* 1\n oof_df[7] = oof_df[7]* 1\n oof_df[8] = oof_df[8]* 1\n oof_df[12] = oof_df[12]* 1\n oof_df[10] = oof_df[10]* 1\n oof_df[13] = oof_df[13]* 1\n\n #This part scales them so their sum makes 1 back again.\n sum = oof_df[[0,1,2,3,4,5,6,7,8,9,10,11,12,13]].sum(axis=1)\n for col in [0,1,2,3,4,5,6,7,8,9,10,11,12,13]:\n oof_df[col] = oof_df[col]/sum\n\n\n return oof_df.values\n\noof_preds_modfy = change(oof_preds) #Give oof preds array to the function\n\nprint(\"Scores OOF Processed: \",multi_weighted_logloss(y_true=y, y_preds=oof_preds_modfy))\nprint(\"Scores OOF Original: \",multi_weighted_logloss(y_true=y, y_preds=oof_preds))\nIt’s quite simple. Try different weights for each class one by one and check your multi weighted logloss if it improved or not. For each class it has one direction for improvement, so it takes a little time to find correct combination. Each model has different characteristics, so you have to adjust them based on your oof. Here is my weights. (Some look weird, doesn’t it? But improved)
\n\noof_df[3] = oof_df[3]*0.55\noof_df[11] = oof_df[11]*0.5\noof_df[0] = oof_df[0]*2\noof_df[1] = oof_df[1]*0.85\noof_df[2] = oof_df[2]*0.5\noof_df[4] = oof_df[4]*1\noof_df[5] = oof_df[5]*4\noof_df[6] = oof_df[6]*0.65\noof_df[7] = oof_df[7]*2.9\noof_df[8] = oof_df[8]*0.35\noof_df[12] = oof_df[12]*3.7\noof_df[10] = oof_df[10]*1.9\noof_df[13] = oof_df[13]*0.9\nMoreover, yesterday we had one left submission and I wanted to try another postprocessing.\nIt was simply this code:
\n\npred[i] = pred[i].apply(lambda x: 1.1 if x>0.76 else x ) \n#After changing preds, make sure that sum of preds make 1 again.\nOf course, first I tried on oof predictions and it was getting improvement in third decimals. So our final submission moved from 0.854 to 0.851 (Private: 0.87021 to 0.86777) and it helped us to move one place up in Private LB :P.
\n\nI would be very happy indeed if you also try and share with us if it worked for you as well or not.
\n\nSee you in next competition!
", "messages": [ { "id": "441617", "postDate": "12/18/2018 20:59:13", "content": "Hi all!\nFirst congrats to all competitors and especially solo gold winners. They just did an amazing job. I especially thank to @ogrellier for his great kernels. First time I’ve used chunk method and multi-threading.
\n\nOur final submission was a blend of our single models and I had a single LGBM model with a CV of 0.493 and LB of 0.915. For me, most important boost came from using ratio of different passband features and using different sample weights instead of direct inverse ratio of class counts. Also using top 55 features as a feature selection method, gave another important boost.
\n\nMy teammates will explain details of models and features in another thread. In this thread I want to mention about postprocessing trick I’ve used and till now did not see in another thread. First I validated postprocessing on my OOF predictions and it also improved my LB score. After I was invited by the team, same process improved their models' both CV and LB.
\n\nOf course, as the model itself improves, gain from postprocess decreases. But, I think it will improve most of your models as well.
\n\nBefore this method, my best model had a CV of 0.52 and LB of 0.968 (Private: 0.99101). With this method CV moved to 0.493 and LB to 0.915 (Private: 0.93077) :)
\n\nI think what lies behind this trick to work is the competition metric. For each row, the correct class’s probability is multiplied with 1 and the rest with zero. That is, it does not matter how do you distribute probabilities among wrong classes. So with this operation, you can compensate the lack of your model by simply changing class preds with weights. For example, my model was giving higher probability for class 42 and class 90 most of the time and lowering their predictions always improved my CV.
\n\nHere is the code for your oof predictions. You can give a try:
\n\ndef change(preds):\n oof_df = pd.DataFrame(preds).copy()\n oof_df[3] = oof_df[3]* 1\n oof_df[11] = oof_df[11]* 1\n oof_df[0] = oof_df[0]* 1\n oof_df[1] = oof_df[1]* 1\n oof_df[2] = oof_df[2]* 1\n oof_df[4] = oof_df[4]* 1\n oof_df[5] = oof_df[5]* 1\n oof_df[6] = oof_df[6]* 1\n oof_df[7] = oof_df[7]* 1\n oof_df[8] = oof_df[8]* 1\n oof_df[12] = oof_df[12]* 1\n oof_df[10] = oof_df[10]* 1\n oof_df[13] = oof_df[13]* 1\n\n #This part scales them so their sum makes 1 back again.\n sum = oof_df[[0,1,2,3,4,5,6,7,8,9,10,11,12,13]].sum(axis=1)\n for col in [0,1,2,3,4,5,6,7,8,9,10,11,12,13]:\n oof_df[col] = oof_df[col]/sum\n\n\n return oof_df.values\n\noof_preds_modfy = change(oof_preds) #Give oof preds array to the function\n\nprint(\"Scores OOF Processed: \",multi_weighted_logloss(y_true=y, y_preds=oof_preds_modfy))\nprint(\"Scores OOF Original: \",multi_weighted_logloss(y_true=y, y_preds=oof_preds))\nIt’s quite simple. Try different weights for each class one by one and check your multi weighted logloss if it improved or not. For each class it has one direction for improvement, so it takes a little time to find correct combination. Each model has different characteristics, so you have to adjust them based on your oof. Here is my weights. (Some look weird, doesn’t it? But improved)
\n\noof_df[3] = oof_df[3]*0.55\noof_df[11] = oof_df[11]*0.5\noof_df[0] = oof_df[0]*2\noof_df[1] = oof_df[1]*0.85\noof_df[2] = oof_df[2]*0.5\noof_df[4] = oof_df[4]*1\noof_df[5] = oof_df[5]*4\noof_df[6] = oof_df[6]*0.65\noof_df[7] = oof_df[7]*2.9\noof_df[8] = oof_df[8]*0.35\noof_df[12] = oof_df[12]*3.7\noof_df[10] = oof_df[10]*1.9\noof_df[13] = oof_df[13]*0.9\nMoreover, yesterday we had one left submission and I wanted to try another postprocessing.\nIt was simply this code:
\n\npred[i] = pred[i].apply(lambda x: 1.1 if x>0.76 else x ) \n#After changing preds, make sure that sum of preds make 1 again.\nOf course, first I tried on oof predictions and it was getting improvement in third decimals. So our final submission moved from 0.854 to 0.851 (Private: 0.87021 to 0.86777) and it helped us to move one place up in Private LB :P.
\n\nI would be very happy indeed if you also try and share with us if it worked for you as well or not.
\n\nSee you in next competition!
", "rawMarkdown": "Hi all!\nFirst congrats to all competitors and especially solo gold winners. They just did an amazing job. I especially thank to @ogrellier for his great kernels. First time I’ve used chunk method and multi-threading.\n\nOur final submission was a blend of our single models and I had a single LGBM model with a CV of 0.493 and LB of 0.915. For me, most important boost came from using ratio of different passband features and using different sample weights instead of direct inverse ratio of class counts. Also using top 55 features as a feature selection method, gave another important boost.\n\nMy teammates will explain details of models and features in another thread. In this thread I want to mention about postprocessing trick I’ve used and till now did not see in another thread. First I validated postprocessing on my OOF predictions and it also improved my LB score. After I was invited by the team, same process improved their models' both CV and LB.\n\nOf course, as the model itself improves, gain from postprocess decreases. But, I think it will improve most of your models as well.\n\nBefore this method, my best model had a CV of 0.52 and LB of 0.968 (Private: 0.99101). With this method CV moved to 0.493 and LB to 0.915 (Private: 0.93077) :)\n\nI think what lies behind this trick to work is the competition metric. For each row, the correct class’s probability is multiplied with 1 and the rest with zero. That is, it does not matter how do you distribute probabilities among wrong classes. So with this operation, you can compensate the lack of your model by simply changing class preds with weights. For example, my model was giving higher probability for class 42 and class 90 most of the time and lowering their predictions always improved my CV. \n\nHere is the code for your oof predictions. You can give a try:\n\n\n\n def change(preds):\n oof_df = pd.DataFrame(preds).copy()\n oof_df[3] = oof_df[3]* 1\n oof_df[11] = oof_df[11]* 1\n oof_df[0] = oof_df[0]* 1\n oof_df[1] = oof_df[1]* 1\n oof_df[2] = oof_df[2]* 1\n oof_df[4] = oof_df[4]* 1\n oof_df[5] = oof_df[5]* 1\n oof_df[6] = oof_df[6]* 1\n oof_df[7] = oof_df[7]* 1\n oof_df[8] = oof_df[8]* 1\n oof_df[12] = oof_df[12]* 1\n oof_df[10] = oof_df[10]* 1\n oof_df[13] = oof_df[13]* 1\n \n \t #This part scales them so their sum makes 1 back again.\n sum = oof_df[[0,1,2,3,4,5,6,7,8,9,10,11,12,13]].sum(axis=1)\n for col in [0,1,2,3,4,5,6,7,8,9,10,11,12,13]:\n oof_df[col] = oof_df[col]/sum\n \n \n return oof_df.values\n \n oof_preds_modfy = change(oof_preds) #Give oof preds array to the function\n \n print(\"Scores OOF Processed: \",multi_weighted_logloss(y_true=y, y_preds=oof_preds_modfy))\n print(\"Scores OOF Original: \",multi_weighted_logloss(y_true=y, y_preds=oof_preds))\n\nIt’s quite simple. Try different weights for each class one by one and check your multi weighted logloss if it improved or not. For each class it has one direction for improvement, so it takes a little time to find correct combination. Each model has different characteristics, so you have to adjust them based on your oof. Here is my weights. (Some look weird, doesn’t it? But improved)\n\n oof_df[3] = oof_df[3]*0.55\n oof_df[11] = oof_df[11]*0.5\n oof_df[0] = oof_df[0]*2\n oof_df[1] = oof_df[1]*0.85\n oof_df[2] = oof_df[2]*0.5\n oof_df[4] = oof_df[4]*1\n oof_df[5] = oof_df[5]*4\n oof_df[6] = oof_df[6]*0.65\n oof_df[7] = oof_df[7]*2.9\n oof_df[8] = oof_df[8]*0.35\n oof_df[12] = oof_df[12]*3.7\n oof_df[10] = oof_df[10]*1.9\n oof_df[13] = oof_df[13]*0.9\n\n\nMoreover, yesterday we had one left submission and I wanted to try another postprocessing.\nIt was simply this code:\n\n pred[i] = pred[i].apply(lambda x: 1.1 if x>0.76 else x ) \n #After changing preds, make sure that sum of preds make 1 again.\n\nOf course, first I tried on oof predictions and it was getting improvement in third decimals. So our final submission moved from 0.854 to 0.851 (Private: 0.87021 to 0.86777) and it helped us to move one place up in Private LB :P.\n\nI would be very happy indeed if you also try and share with us if it worked for you as well or not.\n\nSee you in next competition!", "votes": null }, { "id": "441651", "postDate": "12/18/2018 22:26:47", "content": "Congrats. Nice trick, my CV score improved by 0.03 with this method.
", "rawMarkdown": "Congrats. Nice trick, my CV score improved by 0.03 with this method.", "votes": null }, { "id": "441670", "postDate": "12/18/2018 23:07:33", "content": "Great! What about LB? Did you apply it to corresponding submission?
", "rawMarkdown": "Great! What about LB? Did you apply it to corresponding submission?", "votes": null }, { "id": "441673", "postDate": "12/18/2018 23:17:58", "content": "I did and it improved my private LB score almost by 0.02. Really nice trick.
", "rawMarkdown": "I did and it improved my private LB score almost by 0.02. Really nice trick.", "votes": null } ], "comments": [ { "id": 441651, "author_name": "fatall", "author_url": "", "post_date": "12/18/2018 22:26:47", "content": "Congrats. Nice trick, my CV score improved by 0.03 with this method.
", "votes": null, "replies": [ { "id": 441670, "author_name": "fatihozturk", "author_url": "", "post_date": "12/18/2018 23:07:33", "content": "Great! What about LB? Did you apply it to corresponding submission?
", "votes": null, "replies": [] }, { "id": 441673, "author_name": "fatall", "author_url": "", "post_date": "12/18/2018 23:17:58", "content": "I did and it improved my private LB score almost by 0.02. Really nice trick.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "441617": "Hi all!\nFirst congrats to all competitors and especially solo gold winners. They just did an amazing job. I especially thank to @ogrellier for his great kernels. First time I’ve used chunk method and multi-threading.\n\nOur final submission was a blend of our single models and I had a single LGBM model with a CV of 0.493 and LB of 0.915. For me, most important boost came from using ratio of different passband features and using different sample weights instead of direct inverse ratio of class counts. Also using top 55 features as a feature selection method, gave another important boost.\n\nMy teammates will explain details of models and features in another thread. In this thread I want to mention about postprocessing trick I’ve used and till now did not see in another thread. First I validated postprocessing on my OOF predictions and it also improved my LB score. After I was invited by the team, same process improved their models' both CV and LB.\n\nOf course, as the model itself improves, gain from postprocess decreases. But, I think it will improve most of your models as well.\n\nBefore this method, my best model had a CV of 0.52 and LB of 0.968 (Private: 0.99101). With this method CV moved to 0.493 and LB to 0.915 (Private: 0.93077) :)\n\nI think what lies behind this trick to work is the competition metric. For each row, the correct class’s probability is multiplied with 1 and the rest with zero. That is, it does not matter how do you distribute probabilities among wrong classes. So with this operation, you can compensate the lack of your model by simply changing class preds with weights. For example, my model was giving higher probability for class 42 and class 90 most of the time and lowering their predictions always improved my CV. \n\nHere is the code for your oof predictions. You can give a try:\n\n\n\n def change(preds):\n oof_df = pd.DataFrame(preds).copy()\n oof_df[3] = oof_df[3]* 1\n oof_df[11] = oof_df[11]* 1\n oof_df[0] = oof_df[0]* 1\n oof_df[1] = oof_df[1]* 1\n oof_df[2] = oof_df[2]* 1\n oof_df[4] = oof_df[4]* 1\n oof_df[5] = oof_df[5]* 1\n oof_df[6] = oof_df[6]* 1\n oof_df[7] = oof_df[7]* 1\n oof_df[8] = oof_df[8]* 1\n oof_df[12] = oof_df[12]* 1\n oof_df[10] = oof_df[10]* 1\n oof_df[13] = oof_df[13]* 1\n \n \t #This part scales them so their sum makes 1 back again.\n sum = oof_df[[0,1,2,3,4,5,6,7,8,9,10,11,12,13]].sum(axis=1)\n for col in [0,1,2,3,4,5,6,7,8,9,10,11,12,13]:\n oof_df[col] = oof_df[col]/sum\n \n \n return oof_df.values\n \n oof_preds_modfy = change(oof_preds) #Give oof preds array to the function\n \n print(\"Scores OOF Processed: \",multi_weighted_logloss(y_true=y, y_preds=oof_preds_modfy))\n print(\"Scores OOF Original: \",multi_weighted_logloss(y_true=y, y_preds=oof_preds))\n\nIt’s quite simple. Try different weights for each class one by one and check your multi weighted logloss if it improved or not. For each class it has one direction for improvement, so it takes a little time to find correct combination. Each model has different characteristics, so you have to adjust them based on your oof. Here is my weights. (Some look weird, doesn’t it? But improved)\n\n oof_df[3] = oof_df[3]*0.55\n oof_df[11] = oof_df[11]*0.5\n oof_df[0] = oof_df[0]*2\n oof_df[1] = oof_df[1]*0.85\n oof_df[2] = oof_df[2]*0.5\n oof_df[4] = oof_df[4]*1\n oof_df[5] = oof_df[5]*4\n oof_df[6] = oof_df[6]*0.65\n oof_df[7] = oof_df[7]*2.9\n oof_df[8] = oof_df[8]*0.35\n oof_df[12] = oof_df[12]*3.7\n oof_df[10] = oof_df[10]*1.9\n oof_df[13] = oof_df[13]*0.9\n\n\nMoreover, yesterday we had one left submission and I wanted to try another postprocessing.\nIt was simply this code:\n\n pred[i] = pred[i].apply(lambda x: 1.1 if x>0.76 else x ) \n #After changing preds, make sure that sum of preds make 1 again.\n\nOf course, first I tried on oof predictions and it was getting improvement in third decimals. So our final submission moved from 0.854 to 0.851 (Private: 0.87021 to 0.86777) and it helped us to move one place up in Private LB :P.\n\nI would be very happy indeed if you also try and share with us if it worked for you as well or not.\n\nSee you in next competition!", "441651": "Congrats. Nice trick, my CV score improved by 0.03 with this method.", "441670": "Great! What about LB? Did you apply it to corresponding submission?", "441673": "I did and it improved my private LB score almost by 0.02. Really nice trick." }, "source": "meta" }