![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9718188%2F59c621a5208c26c274b04a2e336c8758%2Fbeforeupdate.png?generation=1665818116598299&alt=media\")
I would like first to thank @ambrosm for pointing out to me the The data update of 2022-09-10 and its implications in the scoring.
\nThis competition is actually 2 competitions in one : Cite and Multiome. We only see the combined score in the leaderboard.
\nHowever we want to know what are the weights for each part in the scoring. Are they balanced? should we spend more time and effort on Multiome or Cite?
\nThe first idea would be to look at the ratio of data in the submission file : the first 6,812,820 values are for Cite and the remaining 58,931,360 values are for Multiome. So we may think at first glance that Multiome is 8.6 time more important than Cite. That is however completely wrong.
\nWhat matters in the scoring is the number of biological cells. Each cell is a unit of scoring and the predictions on each cell are compared to the ground truth through a correlation score.
\nEach cell is a row in the test data. So let's count the rows!
\nHere the situation is complicated by the fact that there was an error in the data at the start of the competition and the error has been fixed: The data update of 2022-09-10
\nWe will start by calculating the weights before the data update. We will then look at the implications of the data update.
1. BEFORE THE DATA UPDATE
\nThe situation before the data update is relatively simple.
\nCite test has 48,663 rows.
\nMultiome test has 55,935 rows but only 30% are used. So Multiome test has 16,780 rows in the score.
\nBefore the data update the weights are as follow :
\nCite : 48,663/(48,663 + 16,780) = 74.4%
\nMultiome : 16,780/(48,663+16,780) = 25.6%
We can see that contrary to what we may think from the submission file, Cite is actually the most important in the scoring with the following weights:
\nCite : 74.4% and Multiome 25.6%
The reason why Cite is more important despite occupying such a small ratio in the submission file is that Mutiome test has much longer rows. There are 3,512 columns after sampling in Multiome test while there are only 140 columns in Cite test. Multiome takes a lot of the submission file because it has many columns!
\n2. AFTER THE DATA UPDATE
\nThe data Update had some (unintended) consequences on the weights. Here We will carefully differentiate the Public and Private Leaderboard. We didn't have to do this before the data update as everything was the same in the Public and Private Leaderboard. But this is no longer the case.
\nIf you carefully read The data update of 2022-09-10 you notice that there is now 7,476 rows in Cite Test Public which are ignored. The key word here is Public. Regarding the Private Test, nothing changed and the weights we calculated are still valid.
\nThe Public/Private split is approximately 42%/58%
Let's first look at the Private numbers :
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9718188%2F85fb582b2f8fbf39ef39326bc745b3d0%2Fprivateafterupdate.png?generation=1665823538853702&alt=media\")
We can see that the ratios are unchanged after the data update:
\nPrivate LB: Cite 74.4% and Multiome 25.6%
Let's now look at the Public numbers:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9718188%2F14b5185761a8bbd6b97a9b6ed1bb73a0%2Fpublicafterupdate.png?generation=1665823568408391&alt=media\")
\nThere are 7,476 rows in the public Cite which are ignored in the scoring. This reduces the importance of Cite and the ratios are :
\nPublic LB: Cite 64.8% and Multiome 35.2%
CONCLUSION
\nThe weights before and after the data update are as follow:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9718188%2F3e33995150dc09d104aa73fabcc905a0%2FSummary.png?generation=1665820339125058&alt=media\")
There are 2 main implications.
\nCite is more important in the Private Leaderbord than in the Public.
\n If your Cite model is better than your Multiome (relatively to everybody else), expect a Shakeup. If your Multiome is better than your Cite, expect a Shakedown.
The scores in the Private Leaderboard will be higher than the score in the Public Leaderboard.
\nSuppose your Cite score is 89.4 and your Multiome score is 66.5 (these values are reasonable if you look at your CV scores)
\nThen your Public and Private scores will be as follow:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9718188%2F3abdac0e960f9330f540b0cf73300e78%2FScoreExample.png?generation=1665821816330587&alt=media\")
There was a drop in the scores after the update. This is due to 2 reasons.
\nI would like first to thank @ambrosm for pointing out to me the The data update of 2022-09-10 and its implications in the scoring.
\nThis competition is actually 2 competitions in one : Cite and Multiome. We only see the combined score in the leaderboard.
\nHowever we want to know what are the weights for each part in the scoring. Are they balanced? should we spend more time and effort on Multiome or Cite?
\nThe first idea would be to look at the ratio of data in the submission file : the first 6,812,820 values are for Cite and the remaining 58,931,360 values are for Multiome. So we may think at first glance that Multiome is 8.6 time more important than Cite. That is however completely wrong.
\nWhat matters in the scoring is the number of biological cells. Each cell is a unit of scoring and the predictions on each cell are compared to the ground truth through a correlation score.
\nEach cell is a row in the test data. So let's count the rows!
\nHere the situation is complicated by the fact that there was an error in the data at the start of the competition and the error has been fixed: The data update of 2022-09-10
\nWe will start by calculating the weights before the data update. We will then look at the implications of the data update.
1. BEFORE THE DATA UPDATE
\nThe situation before the data update is relatively simple.
\nCite test has 48,663 rows.
\nMultiome test has 55,935 rows but only 30% are used. So Multiome test has 16,780 rows in the score.
\nBefore the data update the weights are as follow :
\nCite : 48,663/(48,663 + 16,780) = 74.4%
\nMultiome : 16,780/(48,663+16,780) = 25.6%
We can see that contrary to what we may think from the submission file, Cite is actually the most important in the scoring with the following weights:
\nCite : 74.4% and Multiome 25.6%
The reason why Cite is more important despite occupying such a small ratio in the submission file is that Mutiome test has much longer rows. There are 3,512 columns after sampling in Multiome test while there are only 140 columns in Cite test. Multiome takes a lot of the submission file because it has many columns!
\n2. AFTER THE DATA UPDATE
\nThe data Update had some (unintended) consequences on the weights. Here We will carefully differentiate the Public and Private Leaderboard. We didn't have to do this before the data update as everything was the same in the Public and Private Leaderboard. But this is no longer the case.
\nIf you carefully read The data update of 2022-09-10 you notice that there is now 7,476 rows in Cite Test Public which are ignored. The key word here is Public. Regarding the Private Test, nothing changed and the weights we calculated are still valid.
\nThe Public/Private split is approximately 42%/58%
Let's first look at the Private numbers :
\n
We can see that the ratios are unchanged after the data update:
\nPrivate LB: Cite 74.4% and Multiome 25.6%
Let's now look at the Public numbers:
\n
\nThere are 7,476 rows in the public Cite which are ignored in the scoring. This reduces the importance of Cite and the ratios are :
\nPublic LB: Cite 64.8% and Multiome 35.2%
CONCLUSION
\nThe weights before and after the data update are as follow:
\n
There are 2 main implications.
\nCite is more important in the Private Leaderbord than in the Public.
\n If your Cite model is better than your Multiome (relatively to everybody else), expect a Shakeup. If your Multiome is better than your Cite, expect a Shakedown.
The scores in the Private Leaderboard will be higher than the score in the Public Leaderboard.
\nSuppose your Cite score is 89.4 and your Multiome score is 66.5 (these values are reasonable if you look at your CV scores)
\nThen your Public and Private scores will be as follow:
\n
There was a drop in the scores after the update. This is due to 2 reasons.
\n@gehallak this is super helpful a novice question can u tell me the math on how u reached 81.33 :D . I am just not getting that number
", "rawMarkdown": "gehallak this is super helpful a novice question can u tell me the math on how u reached 81.33 :D . I am just not getting that number", "votes": null }, { "id": "1990857", "postDate": "10/16/2022 19:00:11", "content": "Hi @gauravbrills . I am glad you found it useful. I applied the weights. 64.8% for Cite and 35.2% for Multiome.
\n89.40 * 0.648+ 66.50 * 0.352
\nTo be more precise, with one more decimal: 89.40 * 0.6478 + 66.50 * 0.3522=81.33
Ahh me silly multiplied in the wrong ratio 😃 thanks
", "rawMarkdown": "Ahh me silly multiplied in the wrong ratio 😃 thanks", "votes": null }, { "id": "1991032", "postDate": "10/16/2022 23:08:06", "content": "@gehallak works well now just estimated scores now way better but it does give some correlation . May I ask the sample scores u selected are using which CV strategy ?
", "rawMarkdown": "gehallak works well now just estimated scores now way better but it does give some correlation . May I ask the sample scores u selected are using which CV strategy ?", "votes": null }, { "id": "1991257", "postDate": "10/17/2022 04:23:39", "content": "@gauravbrills, Yes the correlation is good with these weights. For example, from the improvement on my Multiome CV score, I was able to estimate the improvement in the ranking on the LB. It is not easy because we don't see enough decimals on the LB but possible if you suppose the scores are evenly incrementing.
\nI tried Neural Network, Catboost, XGB, LGBM and Linear Regression. But the examples are fictitious.
I suppose CITE weights 0.661, MULTI weights 0.339 in submission correlation score
\nIt is discussed here
@gehallak this is super helpful a novice question can u tell me the math on how u reached 81.33 :D . I am just not getting that number
", "votes": null, "replies": [ { "id": 1990857, "author_name": "gehallak", "author_url": "", "post_date": "10/16/2022 19:00:11", "content": "Hi @gauravbrills . I am glad you found it useful. I applied the weights. 64.8% for Cite and 35.2% for Multiome.
\n89.40 * 0.648+ 66.50 * 0.352
\nTo be more precise, with one more decimal: 89.40 * 0.6478 + 66.50 * 0.3522=81.33
Ahh me silly multiplied in the wrong ratio 😃 thanks
", "votes": null, "replies": [] }, { "id": 1991032, "author_name": "gauravbrills", "author_url": "", "post_date": "10/16/2022 23:08:06", "content": "@gehallak works well now just estimated scores now way better but it does give some correlation . May I ask the sample scores u selected are using which CV strategy ?
", "votes": null, "replies": [] }, { "id": 1991257, "author_name": "gehallak", "author_url": "", "post_date": "10/17/2022 04:23:39", "content": "@gauravbrills, Yes the correlation is good with these weights. For example, from the improvement on my Multiome CV score, I was able to estimate the improvement in the ranking on the LB. It is not easy because we don't see enough decimals on the LB but possible if you suppose the scores are evenly incrementing.
\nI tried Neural Network, Catboost, XGB, LGBM and Linear Regression. But the examples are fictitious.
I suppose CITE weights 0.661, MULTI weights 0.339 in submission correlation score
\nIt is discussed here