{ "id": 420884, "title": "Efficiency: 14th place Public", "url": "/competitions/predict-student-performance-from-game-play/writeups/robert-hatch-efficiency-14th-place-public", "author_name": "", "post_date": "2023-07-03T04:05:23.238589800Z", "votes": 7, "comment_count": 4, "views": 0, "content": "

Code:
\nTraining: https://www.kaggle.com/roberthatch/student-efficiency-catboost-polars-train
\nInference: https://www.kaggle.com/code/roberthatch/student-efficiency-polars-inference

\n

My efficiency solution wasn't very innovative, mainly due to limited time. I only did first couple weeks of competition, then last week of competition, and didn't actually focus too much on efficiency solution, even though it was much more interesting to me, as I wanted a silver medal. (But failed to medal.)

\n

[Aside: yes, I know it was a 5 month competition. Frankly, I was excited to get a second chance after 60 days in which I didn't even edit a single Kaggle Notebook. But I also split my time on the second chance sign language competition, so only gave myself a week on this one.]

\n

The interesting thing was taking @carnozhao 's excellent Polars starter notebook and fixing the inference script to allow submission with latest API was already 15th place public(!)

\n

So - on public LB - I only went from 15th ->14th place with some small improvements. With private LB I probably gained another few places, as public score was 682->687, but private score was 681->691. Given that it took about 2.5 more minutes, that makes the public vs private even more of a difference, at about +0.0025 vs +0.0075, so much more improvement on private LB.

\n

The semi-minimal updates to get that from 0.682 -> 0.687, while adding about 140 seconds to the runtime.

\n\n

Many, many things didn't have time to implement and try:

\n", "messages": [ { "id": "2327590", "postDate": "07/03/2023 04:05:23", "content": "

Code:
\nTraining: https://www.kaggle.com/roberthatch/student-efficiency-catboost-polars-train
\nInference: https://www.kaggle.com/code/roberthatch/student-efficiency-polars-inference

\n

My efficiency solution wasn't very innovative, mainly due to limited time. I only did first couple weeks of competition, then last week of competition, and didn't actually focus too much on efficiency solution, even though it was much more interesting to me, as I wanted a silver medal. (But failed to medal.)

\n

[Aside: yes, I know it was a 5 month competition. Frankly, I was excited to get a second chance after 60 days in which I didn't even edit a single Kaggle Notebook. But I also split my time on the second chance sign language competition, so only gave myself a week on this one.]

\n

The interesting thing was taking @carnozhao 's excellent Polars starter notebook and fixing the inference script to allow submission with latest API was already 15th place public(!)

\n

So - on public LB - I only went from 15th ->14th place with some small improvements. With private LB I probably gained another few places, as public score was 682->687, but private score was 681->691. Given that it took about 2.5 more minutes, that makes the public vs private even more of a difference, at about +0.0025 vs +0.0075, so much more improvement on private LB.

\n

The semi-minimal updates to get that from 0.682 -> 0.687, while adding about 140 seconds to the runtime.

\n\n

Many, many things didn't have time to implement and try:

\n", "rawMarkdown": "Code:\nTraining: https://www.kaggle.com/roberthatch/student-efficiency-catboost-polars-train\nInference: https://www.kaggle.com/code/roberthatch/student-efficiency-polars-inference\n\nMy efficiency solution wasn't very innovative, mainly due to limited time. I only did first couple weeks of competition, then last week of competition, and didn't actually focus too much on efficiency solution, even though it was much more interesting to me, as I wanted a silver medal. (But failed to medal.)\n\n[Aside: yes, I know it was a 5 month competition. Frankly, I was excited to get a second chance after 60 days in which I didn't even edit a single Kaggle Notebook. But I also split my time on the second chance sign language competition, so only gave myself a week on this one.]\n\nThe interesting thing was taking @carnozhao 's excellent [Polars starter notebook](https://www.kaggle.com/code/carnozhao/cpu-catboost-baseline-using-polars-inference) and fixing the inference script to allow submission with latest API was already 15th place public(!)\n\nSo - on public LB - I only went from 15th ->14th place with some small improvements. With private LB I probably gained another few places, as public score was 682->687, but private score was 681->691. Given that it took about 2.5 more minutes, that makes the public vs private even more of a difference, at about +0.0025 vs +0.0075, so much more improvement on private LB.\n\nThe semi-minimal updates to get that from 0.682 -> 0.687, while adding about 140 seconds to the runtime.\n* 3 digit threshold.\n* So far: [no CV test, 0.681 public, 0.682 private]\n* Rerun training with all data\n* SKIP 6 questions (inspired by top public notebook). Skip 2, 3, 12, 13, 16, 18. (Guess 0 on Q13, 1 on the rest.) Naturally this speeds up inference, which is why I decided to be slightly aggressive on how many questions to skip. Only about -0.0001 CV penalty each for skipping the 4th, 5th, 6th questions. It gives an added benefit that I could get lucky and if a question predicts poorly on private LB, (since private LB is a different population), then I would avoid a score penalty that would affect most everyone else.\n* With all the above so far: [0.6819 CV, 0.682 public, 0.683 private, ~5:25 scoring time]\n* 34->60 features [0.6859 CV, 0.685 public, 0.683 private, ~7:15 scoring time run A, ~6:37 run B ]\n* 60->117 features\n* Train on all train data with per question iterations (number of trees) based on results of 5 fold early stopping tuned for each question individually.\n* [0.688 CV, 0.687 public, 0.691 private, ~7:41 scoring time]\n\nMany, many things didn't have time to implement and try:\n* checkpoint features\n* save prior level predictions (or all features)\n* hyper-parameter tuning\n* Extensive feature selection to optimize for efficiency prize\n* Use checkpoint features for CutMix style data augmentation.", "votes": null }, { "id": "2327604", "postDate": "07/03/2023 04:27:00", "content": "

Amazing! I am very excited to know what will be the position in final efficiency leaderboard for both of our notebooks with such a little difference in our estimation. 😄
\ngood luck!

", "rawMarkdown": "Amazing! I am very excited to know what will be the position in final efficiency leaderboard for both of our notebooks with such a little difference in our estimation. 😄 \ngood luck!", "votes": null }, { "id": "2327732", "postDate": "07/03/2023 06:15:40", "content": "
\n

SKIP 6 questions (inspired by top public notebook). Skip 2, 3, 12, 13, 16, 18. (Guess 0 on Q13, 1 on the rest.) Naturally this speeds up inference, which is why I decided to be slightly aggressive on how many questions to skip. Only about -0.0001 CV penalty each for skipping the 4th, 5th, 6th questions. It gives an added benefit that I could get lucky and if a question predicts poorly on private LB, (since private LB is a different population), then I would avoid a score penalty that would affect most everyone else.

\n
\n

Interesting. I tried a similar approach and found, that even predicting Q2 = 1 makes the LB score worse by ~0.002. Since this trick didn't really improve the inference time by a lot (my model was pretty fast anyways), I didn't include it in the final submission.

", "rawMarkdown": ">SKIP 6 questions (inspired by top public notebook). Skip 2, 3, 12, 13, 16, 18. (Guess 0 on Q13, 1 on the rest.) Naturally this speeds up inference, which is why I decided to be slightly aggressive on how many questions to skip. Only about -0.0001 CV penalty each for skipping the 4th, 5th, 6th questions. It gives an added benefit that I could get lucky and if a question predicts poorly on private LB, (since private LB is a different population), then I would avoid a score penalty that would affect most everyone else.\n\nInteresting. I tried a similar approach and found, that even predicting `Q2 = 1` makes the LB score worse by ~0.002. Since this trick didn't really improve the inference time by a lot (my model was pretty fast anyways), I didn't include it in the final submission.", "votes": null }, { "id": "2327780", "postDate": "07/03/2023 07:03:19", "content": "

I figured it would be a gamble, and that public LB score would be pretty random compared with private, so didn't check the difference in LB score if I had predicted all.

", "rawMarkdown": "I figured it would be a gamble, and that public LB score would be pretty random compared with private, so didn't check the difference in LB score if I had predicted all.", "votes": null }, { "id": "2327787", "postDate": "07/03/2023 07:11:37", "content": "

I see. For me, all the submissions have nearly identical public/private scores, but I felt like I was sometimes loosing the CV/LB correlation.

", "rawMarkdown": "I see. For me, all the submissions have nearly identical public/private scores, but I felt like I was sometimes loosing the CV/LB correlation.", "votes": null } ], "comments": [ { "id": 2327604, "author_name": "belalemadhussein", "author_url": "", "post_date": "07/03/2023 04:27:00", "content": "

Amazing! I am very excited to know what will be the position in final efficiency leaderboard for both of our notebooks with such a little difference in our estimation. 😄
\ngood luck!

", "votes": null, "replies": [] }, { "id": 2327732, "author_name": "kononenko", "author_url": "", "post_date": "07/03/2023 06:15:40", "content": "
\n

SKIP 6 questions (inspired by top public notebook). Skip 2, 3, 12, 13, 16, 18. (Guess 0 on Q13, 1 on the rest.) Naturally this speeds up inference, which is why I decided to be slightly aggressive on how many questions to skip. Only about -0.0001 CV penalty each for skipping the 4th, 5th, 6th questions. It gives an added benefit that I could get lucky and if a question predicts poorly on private LB, (since private LB is a different population), then I would avoid a score penalty that would affect most everyone else.

\n
\n

Interesting. I tried a similar approach and found, that even predicting Q2 = 1 makes the LB score worse by ~0.002. Since this trick didn't really improve the inference time by a lot (my model was pretty fast anyways), I didn't include it in the final submission.

", "votes": null, "replies": [ { "id": 2327780, "author_name": "roberthatch", "author_url": "", "post_date": "07/03/2023 07:03:19", "content": "

I figured it would be a gamble, and that public LB score would be pretty random compared with private, so didn't check the difference in LB score if I had predicted all.

", "votes": null, "replies": [ { "id": 2327787, "author_name": "kononenko", "author_url": "", "post_date": "07/03/2023 07:11:37", "content": "

I see. For me, all the submissions have nearly identical public/private scores, but I felt like I was sometimes loosing the CV/LB correlation.

", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2327590": "Code:\nTraining: https://www.kaggle.com/roberthatch/student-efficiency-catboost-polars-train\nInference: https://www.kaggle.com/code/roberthatch/student-efficiency-polars-inference\n\nMy efficiency solution wasn't very innovative, mainly due to limited time. I only did first couple weeks of competition, then last week of competition, and didn't actually focus too much on efficiency solution, even though it was much more interesting to me, as I wanted a silver medal. (But failed to medal.)\n\n[Aside: yes, I know it was a 5 month competition. Frankly, I was excited to get a second chance after 60 days in which I didn't even edit a single Kaggle Notebook. But I also split my time on the second chance sign language competition, so only gave myself a week on this one.]\n\nThe interesting thing was taking @carnozhao 's excellent [Polars starter notebook](https://www.kaggle.com/code/carnozhao/cpu-catboost-baseline-using-polars-inference) and fixing the inference script to allow submission with latest API was already 15th place public(!)\n\nSo - on public LB - I only went from 15th ->14th place with some small improvements. With private LB I probably gained another few places, as public score was 682->687, but private score was 681->691. Given that it took about 2.5 more minutes, that makes the public vs private even more of a difference, at about +0.0025 vs +0.0075, so much more improvement on private LB.\n\nThe semi-minimal updates to get that from 0.682 -> 0.687, while adding about 140 seconds to the runtime.\n* 3 digit threshold.\n* So far: [no CV test, 0.681 public, 0.682 private]\n* Rerun training with all data\n* SKIP 6 questions (inspired by top public notebook). Skip 2, 3, 12, 13, 16, 18. (Guess 0 on Q13, 1 on the rest.) Naturally this speeds up inference, which is why I decided to be slightly aggressive on how many questions to skip. Only about -0.0001 CV penalty each for skipping the 4th, 5th, 6th questions. It gives an added benefit that I could get lucky and if a question predicts poorly on private LB, (since private LB is a different population), then I would avoid a score penalty that would affect most everyone else.\n* With all the above so far: [0.6819 CV, 0.682 public, 0.683 private, ~5:25 scoring time]\n* 34->60 features [0.6859 CV, 0.685 public, 0.683 private, ~7:15 scoring time run A, ~6:37 run B ]\n* 60->117 features\n* Train on all train data with per question iterations (number of trees) based on results of 5 fold early stopping tuned for each question individually.\n* [0.688 CV, 0.687 public, 0.691 private, ~7:41 scoring time]\n\nMany, many things didn't have time to implement and try:\n* checkpoint features\n* save prior level predictions (or all features)\n* hyper-parameter tuning\n* Extensive feature selection to optimize for efficiency prize\n* Use checkpoint features for CutMix style data augmentation.", "2327604": "Amazing! I am very excited to know what will be the position in final efficiency leaderboard for both of our notebooks with such a little difference in our estimation. 😄 \ngood luck!", "2327732": ">SKIP 6 questions (inspired by top public notebook). Skip 2, 3, 12, 13, 16, 18. (Guess 0 on Q13, 1 on the rest.) Naturally this speeds up inference, which is why I decided to be slightly aggressive on how many questions to skip. Only about -0.0001 CV penalty each for skipping the 4th, 5th, 6th questions. It gives an added benefit that I could get lucky and if a question predicts poorly on private LB, (since private LB is a different population), then I would avoid a score penalty that would affect most everyone else.\n\nInteresting. I tried a similar approach and found, that even predicting `Q2 = 1` makes the LB score worse by ~0.002. Since this trick didn't really improve the inference time by a lot (my model was pretty fast anyways), I didn't include it in the final submission.", "2327780": "I figured it would be a gamble, and that public LB score would be pretty random compared with private, so didn't check the difference in LB score if I had predicted all.", "2327787": "I see. For me, all the submissions have nearly identical public/private scores, but I felt like I was sometimes loosing the CV/LB correlation." }, "source": "meta" }