Hi there!
\n\nI have question on score evaluation.\nIf a participant uses distinct numbers 1,2,3,4,5 in each row of submission file, then formula \n$$r_s = 1 - \\dfrac{6\\cdot SquareSum_s}{n(n^2-1)}$$ \ncan be used, and since n=5, then\n$$r_s = 1 - \\dfrac{SquareSum_s}{20}.$$\nIf number of tests is N (N ~ 0.17*274 ~ 44...49), then score is\n$$R = \\frac{1}{N} \\sum_{s=1}^{N} r_s = \\frac{1}{20N} \\sum_{s=1}^{N}(20-SquareSum_s).$$\nBut (!) SquareSum_s is integer in this case, \nso 20*N*R must be integer, right?
\n\nWhen consider scores from 'raw data' archive and multiply each by 20*49 (use N=49), then one will get a lot of (almost) integer numbers, which is expectable; but many half-integers too (like 3.5, 10.5, ...).
\n\nI used distinct 'days' 1,2,3,4,5 in each row of my draft submission file, and have got half-integers sometimes.
\n\nQuestion: Why half-integers occur so often?\n
a - I can't believe that so many participants use non-distinct 'days' in submission files.\n
b - maybe wrong my calculations? where?\n
c - maybe divide by 2N, not N (to get scores in [-0.5, 0.5])? but no, looking at current LB;\n
d - other reasons?
Hi there!
\n\nI have question on score evaluation.\nIf a participant uses distinct numbers 1,2,3,4,5 in each row of submission file, then formula \n$$r_s = 1 - \\dfrac{6\\cdot SquareSum_s}{n(n^2-1)}$$ \ncan be used, and since n=5, then\n$$r_s = 1 - \\dfrac{SquareSum_s}{20}.$$\nIf number of tests is N (N ~ 0.17*274 ~ 44...49), then score is\n$$R = \\frac{1}{N} \\sum_{s=1}^{N} r_s = \\frac{1}{20N} \\sum_{s=1}^{N}(20-SquareSum_s).$$\nBut (!) SquareSum_s is integer in this case, \nso 20*N*R must be integer, right?
\n\nWhen consider scores from 'raw data' archive and multiply each by 20*49 (use N=49), then one will get a lot of (almost) integer numbers, which is expectable; but many half-integers too (like 3.5, 10.5, ...).
\n\nI used distinct 'days' 1,2,3,4,5 in each row of my draft submission file, and have got half-integers sometimes.
\n\nQuestion: Why half-integers occur so often?\n
a - I can't believe that so many participants use non-distinct 'days' in submission files.\n
b - maybe wrong my calculations? where?\n
c - maybe divide by 2N, not N (to get scores in [-0.5, 0.5])? but no, looking at current LB;\n
d - other reasons?
When assume that scores were calculated from N=98 testing entries, then all seems to be correct.
\n\nBut then words "This leaderboard is calculated on approximately 17% of the test data" need to be replaced with "... 35% of the test data".
", "rawMarkdown": "When assume that scores were calculated from N=98 testing entries, then all seems to be correct.\r\n\r\nBut then words \"This leaderboard is calculated on approximately 17% of the test data\" need to be replaced with \"... 35% of the test data\".", "votes": null }, { "id": "124052", "postDate": "06/15/2016 07:47:38", "content": "My estimations show that N is divisible by 14, and most probable value for N is 28.
\n\nThen "This leaderboard is calculated on approximately 17% of the test data" need to be replaced with "... 10% of the test data" (or 11%).
", "rawMarkdown": "My estimations show that N is divisible by 14, and most probable value for N is 28. \r\n\r\nThen \"This leaderboard is calculated on approximately 17% of the test data\" need to be replaced with \"... 10% of the test data\" (or 11%).", "votes": null }, { "id": "124117", "postDate": "06/15/2016 17:10:36", "content": "I wondered about that too. My current hypothesis based on William's comment on this thread\nabout no changes in leaderboard score is that it probably is 17% of the data actually being used but recall that not all sets in the test data are being used. So if N=28 for the public leaderboard, then there are only 164 or 165 sets in the public/private leaderboard combined. This would imply that around 85 sets are in neither. I must admit that I have certain sets that I am hoping are in the 85 so to speak...
", "rawMarkdown": "I wondered about that too. My current hypothesis based on William's comment on this thread\r\nabout [no changes in leaderboard score][1] is that it probably is 17% of the data actually being used but recall that not all sets in the test data are being used. So if N=28 for the public leaderboard, then there are only 164 or 165 sets in the public/private leaderboard combined. This would imply that around 85 sets are in neither. I must admit that I have certain sets that I am hoping are in the 85 so to speak...\r\n\r\n [1]: https://www.kaggle.com/c/draper-satellite-image-chronology/forums/t/20983/no-change-in-submission-score-despite-changes/120166#post120166", "votes": null }, { "id": "124144", "postDate": "06/15/2016 19:45:50", "content": "Why N=28, wouldn't N=42 be more likely?
\n\nWilliam said a small number of sets aren't part of the scoring. If N=28 then, as Chris notes, that leaves circa 85 sets omitted, which doesn't seem small to me.
\n\nIf N=42 and there is 17% coverage then the public/private leaderboard will contain between 241 to 254 sets, omitting anything between 20 to 33 sets (as an aside still seems too many to be classed as small to me).
\n\nAnyhow when you also take into account that;
\n\nThen I think that this points to N=42 and that the omitted sets are the 22 sets that geographically overlap the training data.
", "rawMarkdown": "Why N=28, wouldn't N=42 be more likely?\r\n\r\nWilliam said a small number of sets aren't part of the scoring. If N=28 then, as Chris notes, that leaves circa 85 sets omitted, which doesn't seem small to me.\r\n\r\nIf N=42 and there is 17% coverage then the public/private leaderboard will contain between 241 to 254 sets, omitting anything between 20 to 33 sets (*as an aside still seems too many to be classed as small to me*).\r\n\r\nAnyhow when you also take into account that;\r\n\r\n - there are 2 image clusters where the training and test data overlap\r\n - theses overlap clusters contain 22 images\r\n - my personal experience is that none of these images are in the public leaderboard\r\n\r\nThen I think that this points to N=42 and that the omitted sets are the 22 sets that geographically overlap the training data.", "votes": null }, { "id": "124348", "postDate": "06/17/2016 12:06:48", "content": "@Chippy, Yes, the ratio 42/274 is closer to 0.17 than 28/274.
\nAnd I agree that none of these 22 (or 23) images are in the Public LB.
My thought was: since 14| N, then I focused on 14, 28, 42, 56, ... and obtained contradictions for all these values but 28. (I hope my estimations were error-free).
\n\nWell, I just assume that N=28 (don't claim by 100%).
\n\nNon-direct way:\nfocus on any of 5 southern clusters; and try to evaluate number of images which change Private LB score; for each of these clusters I've got ratio \n$$\nr_{cluster} = \\dfrac{images\\; which \\; change\\;Private\\;LB\\; score}{total \\;images} < 0.148;\n$$\nBut in the case of N=42 -- at least one of these clusters must have this ratio greater than\n(42/274) ~ 0.153 .
\n\nAnd there is no guarantee that the rate of ignored images is the same for Private/Public LBs.\n
For example:
\n - Public set: 47 images (28 essential + 19 ignored);
\n - Private set: 227 images (135 essential + 92 ignored).
\nkeeps ratio (more-less).
But it can be
\n - Public set: 47 images (28 essential + 19 ignored);
\n - Private set: 227 images (227 essential + 0 ignored).
\nwhy not?
Here is my main confuse.\nI thought before, that set distribution is extremely easy:\n$$\nPublic Set (17\\%) + Private Set(83\\%) = 274,$$\nbut it seems that it is more complicated.
", "rawMarkdown": "Chippy, Yes, the ratio 42/274 is closer to 0.17 than 28/274.@Virtuoso, thanks for your follow up explanation.
\n\nIt's helpful to know that this is your conclusion based upon more extensive leaderboard feedback than I've been able to do.
\n\nIn my limited leaderboard feedback I've certainly seen strange behaviour in some clusters which would be consistent with your hypothesis.
\n\nTo date though I've thought this behaviour is been more likely to be due to errors in my submissions! Maybe they are right and I'm focusing my efforts in the wrong places?
", "rawMarkdown": "Virtuoso, thanks for your follow up explanation. \r\n\r\nIt's helpful to know that this is your conclusion based upon more extensive leaderboard feedback than I've been able to do. \r\n\r\nIn my limited leaderboard feedback I've certainly seen strange behaviour in some clusters which would be consistent with your hypothesis. \r\n\r\nTo date though I've thought this behaviour is been more likely to be due to errors in my submissions! Maybe they are right and I'm focusing my efforts in the wrong places?", "votes": null }, { "id": "124655", "postDate": "06/20/2016 22:43:10", "content": "@Virtuoso, from the change in my score from a chance submission in which I only changed the order of 1 set I too believe that indeed only 28 sets are contributing to the public leaderboard..!
", "rawMarkdown": "Virtuoso, from the change in my score from a chance submission in which I only changed the order of 1 set I too believe that indeed only 28 sets are contributing to the public leaderboard..!", "votes": null } ], "comments": [ { "id": 119706, "author_name": "virtuoso", "author_url": "", "post_date": "05/12/2016 12:04:27", "content": "When assume that scores were calculated from N=98 testing entries, then all seems to be correct.
\n\nBut then words "This leaderboard is calculated on approximately 17% of the test data" need to be replaced with "... 35% of the test data".
", "votes": null, "replies": [] }, { "id": 124052, "author_name": "virtuoso", "author_url": "", "post_date": "06/15/2016 07:47:38", "content": "My estimations show that N is divisible by 14, and most probable value for N is 28.
\n\nThen "This leaderboard is calculated on approximately 17% of the test data" need to be replaced with "... 10% of the test data" (or 11%).
", "votes": null, "replies": [] }, { "id": 124117, "author_name": "hurlburt", "author_url": "", "post_date": "06/15/2016 17:10:36", "content": "I wondered about that too. My current hypothesis based on William's comment on this thread\nabout no changes in leaderboard score is that it probably is 17% of the data actually being used but recall that not all sets in the test data are being used. So if N=28 for the public leaderboard, then there are only 164 or 165 sets in the public/private leaderboard combined. This would imply that around 85 sets are in neither. I must admit that I have certain sets that I am hoping are in the 85 so to speak...
", "votes": null, "replies": [] }, { "id": 124144, "author_name": "nigelcarpenter", "author_url": "", "post_date": "06/15/2016 19:45:50", "content": "Why N=28, wouldn't N=42 be more likely?
\n\nWilliam said a small number of sets aren't part of the scoring. If N=28 then, as Chris notes, that leaves circa 85 sets omitted, which doesn't seem small to me.
\n\nIf N=42 and there is 17% coverage then the public/private leaderboard will contain between 241 to 254 sets, omitting anything between 20 to 33 sets (as an aside still seems too many to be classed as small to me).
\n\nAnyhow when you also take into account that;
\n\nThen I think that this points to N=42 and that the omitted sets are the 22 sets that geographically overlap the training data.
", "votes": null, "replies": [] }, { "id": 124348, "author_name": "virtuoso", "author_url": "", "post_date": "06/17/2016 12:06:48", "content": "@Chippy, Yes, the ratio 42/274 is closer to 0.17 than 28/274.
\nAnd I agree that none of these 22 (or 23) images are in the Public LB.
My thought was: since 14| N, then I focused on 14, 28, 42, 56, ... and obtained contradictions for all these values but 28. (I hope my estimations were error-free).
\n\nWell, I just assume that N=28 (don't claim by 100%).
\n\nNon-direct way:\nfocus on any of 5 southern clusters; and try to evaluate number of images which change Private LB score; for each of these clusters I've got ratio \n$$\nr_{cluster} = \\dfrac{images\\; which \\; change\\;Private\\;LB\\; score}{total \\;images} < 0.148;\n$$\nBut in the case of N=42 -- at least one of these clusters must have this ratio greater than\n(42/274) ~ 0.153 .
\n\nAnd there is no guarantee that the rate of ignored images is the same for Private/Public LBs.\n
For example:
\n - Public set: 47 images (28 essential + 19 ignored);
\n - Private set: 227 images (135 essential + 92 ignored).
\nkeeps ratio (more-less).
But it can be
\n - Public set: 47 images (28 essential + 19 ignored);
\n - Private set: 227 images (227 essential + 0 ignored).
\nwhy not?
Here is my main confuse.\nI thought before, that set distribution is extremely easy:\n$$\nPublic Set (17\\%) + Private Set(83\\%) = 274,$$\nbut it seems that it is more complicated.
", "votes": null, "replies": [] }, { "id": 124373, "author_name": "nigelcarpenter", "author_url": "", "post_date": "06/17/2016 16:23:19", "content": "@Virtuoso, thanks for your follow up explanation.
\n\nIt's helpful to know that this is your conclusion based upon more extensive leaderboard feedback than I've been able to do.
\n\nIn my limited leaderboard feedback I've certainly seen strange behaviour in some clusters which would be consistent with your hypothesis.
\n\nTo date though I've thought this behaviour is been more likely to be due to errors in my submissions! Maybe they are right and I'm focusing my efforts in the wrong places?
", "votes": null, "replies": [] }, { "id": 124655, "author_name": "nigelcarpenter", "author_url": "", "post_date": "06/20/2016 22:43:10", "content": "@Virtuoso, from the change in my score from a chance submission in which I only changed the order of 1 set I too believe that indeed only 28 sets are contributing to the public leaderboard..!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "119664": "Hi there!\r\n\r\nI have question on score evaluation.\r\nIf a participant uses distinct numbers 1,2,3,4,5 in each row of submission file, then formula \r\n$$r_s = 1 - \\dfrac{6\\cdot SquareSum_s}{n(n^2-1)}$$ \r\ncan be used, and since n=5, then\r\n$$r_s = 1 - \\dfrac{SquareSum_s}{20}.$$\r\nIf number of tests is N (N ~ 0.17*274 ~ 44...49), then score is\r\n$$R = \\frac{1}{N} \\sum_{s=1}^{N} r_s = \\frac{1}{20N} \\sum_{s=1}^{N}(20-SquareSum_s).$$\r\nBut (!) SquareSum_s is integer in this case, \r\nso 20*N*R must be integer, right?\r\n\r\nWhen consider scores from 'raw data' archive and multiply each by 20*49 (use N=49), then one will get a lot of (almost) integer numbers, which is expectable; but *many half-integers too* (like 3.5, 10.5, ...).\r\n\r\nI used distinct 'days' 1,2,3,4,5 in each row of my draft submission file, and have got half-integers sometimes.\r\n\r\n**Question**: *Why half-integers occur so often?*\r\n