{ "id": 451472, "title": "CV and LB are not consistent", "url": "/competitions/predict-ai-model-runtime/discussion/451472", "author_name": "", "post_date": "2023-10-29T01:09:22.585763Z", "votes": 4, "comment_count": 4, "views": 0, "content": "

I'm confused about my CV and LB results. Is everyone's CV and LB consistent?
\n\"\"

", "messages": [ { "id": "2503322", "postDate": "10/29/2023 01:09:22", "content": "

I'm confused about my CV and LB results. Is everyone's CV and LB consistent?
\n\"\"

", "rawMarkdown": "I'm confused about my CV and LB results. Is everyone's CV and LB consistent?\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16620895%2F64d2d48180307e8cd73dc0ecf1a143dd%2FSnipaste_2023-10-29_09-09-03.png?generation=1698541759332741&alt=media)", "votes": null }, { "id": "2503598", "postDate": "10/29/2023 08:41:14", "content": "

It seems like you need to aim for a lower OPA.
\nJust kidding, I believe you should measure your Kendall-tau score instead of your OPA because that is the metric of the competition. It might make a change.

", "rawMarkdown": "It seems like you need to aim for a lower OPA.\nJust kidding, I believe you should measure your Kendall-tau score instead of your OPA because that is the metric of the competition. It might make a change.", "votes": null }, { "id": "2503720", "postDate": "10/29/2023 11:43:57", "content": "

As mentioned in this link https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/448367#2490568, OPA and KT are actually the same. Maybe I sampled too small configs.

", "rawMarkdown": "As mentioned in this link https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/448367#2490568, OPA and KT are actually the same. Maybe I sampled too small configs.", "votes": null }, { "id": "2503727", "postDate": "10/29/2023 11:50:25", "content": "

Yeah, I've seen that. For me, the OPA and Kendal are not exactly the same… I don't know if my implementation is wrong or if whatever he wrote is wrong.

", "rawMarkdown": "Yeah, I've seen that. For me, the OPA and Kendal are not exactly the same... I don't know if my implementation is wrong or if whatever he wrote is wrong.", "votes": null }, { "id": "2503965", "postDate": "10/29/2023 15:07:14", "content": "

@chitra077 it is possible that you didn't sample enough configs however, i suspect that this is a normal aspect of the competition because there are so many configs that you effectively have to optimize for config selection and I suspect there isn't a lot of need to fiddle with the reference implementation aside from that.

", "rawMarkdown": "chitra077 it is possible that you didn't sample enough configs however, i suspect that this is a normal aspect of the competition because there are so many configs that you effectively have to optimize for config selection and I suspect there isn't a lot of need to fiddle with the reference implementation aside from that.", "votes": null } ], "comments": [ { "id": 2503598, "author_name": "amitaharoni", "author_url": "", "post_date": "10/29/2023 08:41:14", "content": "

It seems like you need to aim for a lower OPA.
\nJust kidding, I believe you should measure your Kendall-tau score instead of your OPA because that is the metric of the competition. It might make a change.

", "votes": null, "replies": [ { "id": 2503720, "author_name": "chitra077", "author_url": "", "post_date": "10/29/2023 11:43:57", "content": "

As mentioned in this link https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/448367#2490568, OPA and KT are actually the same. Maybe I sampled too small configs.

", "votes": null, "replies": [ { "id": 2503727, "author_name": "amitaharoni", "author_url": "", "post_date": "10/29/2023 11:50:25", "content": "

Yeah, I've seen that. For me, the OPA and Kendal are not exactly the same… I don't know if my implementation is wrong or if whatever he wrote is wrong.

", "votes": null, "replies": [ { "id": 2503965, "author_name": "rob1080ti", "author_url": "", "post_date": "10/29/2023 15:07:14", "content": "

@chitra077 it is possible that you didn't sample enough configs however, i suspect that this is a normal aspect of the competition because there are so many configs that you effectively have to optimize for config selection and I suspect there isn't a lot of need to fiddle with the reference implementation aside from that.

", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2503322": "I'm confused about my CV and LB results. Is everyone's CV and LB consistent?\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16620895%2F64d2d48180307e8cd73dc0ecf1a143dd%2FSnipaste_2023-10-29_09-09-03.png?generation=1698541759332741&alt=media)", "2503598": "It seems like you need to aim for a lower OPA.\nJust kidding, I believe you should measure your Kendall-tau score instead of your OPA because that is the metric of the competition. It might make a change.", "2503720": "As mentioned in this link https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/448367#2490568, OPA and KT are actually the same. Maybe I sampled too small configs.", "2503727": "Yeah, I've seen that. For me, the OPA and Kendal are not exactly the same... I don't know if my implementation is wrong or if whatever he wrote is wrong.", "2503965": "chitra077 it is possible that you didn't sample enough configs however, i suspect that this is a normal aspect of the competition because there are so many configs that you effectively have to optimize for config selection and I suspect there isn't a lot of need to fiddle with the reference implementation aside from that." }, "source": "meta" }