{ "id": 357785, "title": "CV/LB correlation", "url": "/competitions/open-problems-multimodal/discussion/357785", "author_name": "", "post_date": "2022-10-05T14:01:44.703901200Z", "votes": 6, "comment_count": 9, "views": 0, "content": "

I see a healthy correlation in the Cite part between the CV and the LB submissions. (LB goes up when CV goes up even if not perfectly linearly)
\nBut The Multiome part is not so nice. sometimes CV goes up noticeably and LB goes down (with unchanged Cite part)
\nAlso the ensembling in Multiome doesn't bring much improvement.
\nWould you mind sharing if you see the same ?

", "messages": [ { "id": "1973186", "postDate": "10/05/2022 14:01:44", "content": "

I see a healthy correlation in the Cite part between the CV and the LB submissions. (LB goes up when CV goes up even if not perfectly linearly)
\nBut The Multiome part is not so nice. sometimes CV goes up noticeably and LB goes down (with unchanged Cite part)
\nAlso the ensembling in Multiome doesn't bring much improvement.
\nWould you mind sharing if you see the same ?

", "rawMarkdown": "I see a healthy correlation in the Cite part between the CV and the LB submissions. (LB goes up when CV goes up even if not perfectly linearly)\nBut The Multiome part is not so nice. sometimes CV goes up noticeably and LB goes down (with unchanged Cite part)\nAlso the ensembling in Multiome doesn't bring much improvement.\nWould you mind sharing if you see the same ?", "votes": null }, { "id": "1977387", "postDate": "10/08/2022 02:23:10", "content": "

My result is just the opposite of yours. The improvement of the CV score of the cite part often does not lead to the improvement of the LB, but the improvement of the CV score of the multiome part is reliable.

", "rawMarkdown": "My result is just the opposite of yours. The improvement of the CV score of the cite part often does not lead to the improvement of the LB, but the improvement of the CV score of the multiome part is reliable.", "votes": null }, { "id": "1979938", "postDate": "10/09/2022 20:28:29", "content": "

Take a look:
\nhttps://www.kaggle.com/competitions/open-problems-multimodal/discussion/358860

", "rawMarkdown": "Take a look:\nhttps://www.kaggle.com/competitions/open-problems-multimodal/discussion/358860", "votes": null }, { "id": "1981313", "postDate": "10/10/2022 18:13:16", "content": "

Thank you Alexander, you published many original and interesting ideas. I always read you with great interest.

", "rawMarkdown": "Thank you Alexander, you published many original and interesting ideas. I always read you with great interest.", "votes": null }, { "id": "1981317", "postDate": "10/10/2022 18:17:08", "content": "

thank you @qqzzxxdd for sharing and congratulations, you are doing extremely well !!

", "rawMarkdown": "thank you @qqzzxxdd for sharing and congratulations, you are doing extremely well !!", "votes": null }, { "id": "1999494", "postDate": "10/22/2022 11:50:41", "content": "

My result is alsmost same with you. Cite CV goes up with LB.

", "rawMarkdown": "My result is alsmost same with you. Cite CV goes up with LB.", "votes": null }, { "id": "2001376", "postDate": "10/24/2022 02:05:27", "content": "

Thank you, my Multi is more stable now and more predictable. I also look at the correlation between the single models. The lower the correlation the better the ensembling.

", "rawMarkdown": "Thank you, my Multi is more stable now and more predictable. I also look at the correlation between the single models. The lower the correlation the better the ensembling.", "votes": null }, { "id": "2007397", "postDate": "10/28/2022 07:26:21", "content": "

Thank you for your comment. I can also see a relative healthy corr between CV and LB in Multi.
\nDo you compute corr between Submision file A (from model A) and Submission file B (from model B) ??

", "rawMarkdown": "Thank you for your comment. I can also see a relative healthy corr between CV and LB in Multi. \nDo you compute corr between Submision file A (from model A) and Submission file B (from model B) ??", "votes": null }, { "id": "2007604", "postDate": "10/28/2022 11:06:42", "content": "

yes, one correlation for the Cite part and one for the Multi Part. This is how I do it:

\n
    \n
  1. Read 2 submission files (from models A and B) and merge them by row_id
    2. \n
  2. Add cell id for each row from evaluation_ids file
    3. \n
  3. Separate Cite part and Multi Part (the cut is at 6812820)
    \nFor each part:
    4. \n
  4. GroupBy cell_Id
    5. \n
  5. Calculate correlation between 2 submissions for each cell_id
    6. \n
  6. Calculate the average of the correlations
    7. \n
\n

This way you have 2 correlations between the 2 submissions. One for the Cite part and one for the Multi part.
\nIn my case they are between 98% and 99.5%. Not diversified enough I am afraid.

", "rawMarkdown": "yes, one correlation for the Cite part and one for the Multi Part. This is how I do it:\n\n1. Read 2 submission files (from models A and B) and merge them by row_id\n2. Add cell id for each row from evaluation_ids file\n3. Separate Cite part and Multi Part (the cut is at 6812820)\nFor each part:\n4. GroupBy cell_Id\n5. Calculate correlation between 2 submissions for each cell_id\n6. Calculate the average of the correlations\n\nThis way you have 2 correlations between the 2 submissions. One for the Cite part and one for the Multi part.\nIn my case they are between 98% and 99.5%. Not diversified enough I am afraid.", "votes": null }, { "id": "2008604", "postDate": "10/29/2022 07:57:38", "content": "

Thanks a lot!!

", "rawMarkdown": "Thanks a lot!!", "votes": null } ], "comments": [ { "id": 1977387, "author_name": "qqzzxxdd", "author_url": "", "post_date": "10/08/2022 02:23:10", "content": "

My result is just the opposite of yours. The improvement of the CV score of the cite part often does not lead to the improvement of the LB, but the improvement of the CV score of the multiome part is reliable.

", "votes": null, "replies": [ { "id": 1981317, "author_name": "gehallak", "author_url": "", "post_date": "10/10/2022 18:17:08", "content": "

thank you @qqzzxxdd for sharing and congratulations, you are doing extremely well !!

", "votes": null, "replies": [] } ] }, { "id": 1979938, "author_name": "alexandervc", "author_url": "", "post_date": "10/09/2022 20:28:29", "content": "

Take a look:
\nhttps://www.kaggle.com/competitions/open-problems-multimodal/discussion/358860

", "votes": null, "replies": [ { "id": 1981313, "author_name": "gehallak", "author_url": "", "post_date": "10/10/2022 18:13:16", "content": "

Thank you Alexander, you published many original and interesting ideas. I always read you with great interest.

", "votes": null, "replies": [] } ] }, { "id": 1999494, "author_name": "aesoptacit", "author_url": "", "post_date": "10/22/2022 11:50:41", "content": "

My result is alsmost same with you. Cite CV goes up with LB.

", "votes": null, "replies": [ { "id": 2001376, "author_name": "gehallak", "author_url": "", "post_date": "10/24/2022 02:05:27", "content": "

Thank you, my Multi is more stable now and more predictable. I also look at the correlation between the single models. The lower the correlation the better the ensembling.

", "votes": null, "replies": [] }, { "id": 2007397, "author_name": "aesoptacit", "author_url": "", "post_date": "10/28/2022 07:26:21", "content": "

Thank you for your comment. I can also see a relative healthy corr between CV and LB in Multi.
\nDo you compute corr between Submision file A (from model A) and Submission file B (from model B) ??

", "votes": null, "replies": [] }, { "id": 2007604, "author_name": "gehallak", "author_url": "", "post_date": "10/28/2022 11:06:42", "content": "

yes, one correlation for the Cite part and one for the Multi Part. This is how I do it:

\n
    \n
  1. Read 2 submission files (from models A and B) and merge them by row_id
    2. \n
  2. Add cell id for each row from evaluation_ids file
    3. \n
  3. Separate Cite part and Multi Part (the cut is at 6812820)
    \nFor each part:
    4. \n
  4. GroupBy cell_Id
    5. \n
  5. Calculate correlation between 2 submissions for each cell_id
    6. \n
  6. Calculate the average of the correlations
    7. \n
\n

This way you have 2 correlations between the 2 submissions. One for the Cite part and one for the Multi part.
\nIn my case they are between 98% and 99.5%. Not diversified enough I am afraid.

", "votes": null, "replies": [] }, { "id": 2008604, "author_name": "aesoptacit", "author_url": "", "post_date": "10/29/2022 07:57:38", "content": "

Thanks a lot!!

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1973186": "I see a healthy correlation in the Cite part between the CV and the LB submissions. (LB goes up when CV goes up even if not perfectly linearly)\nBut The Multiome part is not so nice. sometimes CV goes up noticeably and LB goes down (with unchanged Cite part)\nAlso the ensembling in Multiome doesn't bring much improvement.\nWould you mind sharing if you see the same ?", "1977387": "My result is just the opposite of yours. The improvement of the CV score of the cite part often does not lead to the improvement of the LB, but the improvement of the CV score of the multiome part is reliable.", "1979938": "Take a look:\nhttps://www.kaggle.com/competitions/open-problems-multimodal/discussion/358860", "1981313": "Thank you Alexander, you published many original and interesting ideas. I always read you with great interest.", "1981317": "thank you @qqzzxxdd for sharing and congratulations, you are doing extremely well !!", "1999494": "My result is alsmost same with you. Cite CV goes up with LB.", "2001376": "Thank you, my Multi is more stable now and more predictable. I also look at the correlation between the single models. The lower the correlation the better the ensembling.", "2007397": "Thank you for your comment. I can also see a relative healthy corr between CV and LB in Multi. \nDo you compute corr between Submision file A (from model A) and Submission file B (from model B) ??", "2007604": "yes, one correlation for the Cite part and one for the Multi Part. This is how I do it:\n\n1. Read 2 submission files (from models A and B) and merge them by row_id\n2. Add cell id for each row from evaluation_ids file\n3. Separate Cite part and Multi Part (the cut is at 6812820)\nFor each part:\n4. GroupBy cell_Id\n5. Calculate correlation between 2 submissions for each cell_id\n6. Calculate the average of the correlations\n\nThis way you have 2 correlations between the 2 submissions. One for the Cite part and one for the Multi part.\nIn my case they are between 98% and 99.5%. Not diversified enough I am afraid.", "2008604": "Thanks a lot!!" }, "source": "meta" }