We, Team Bruce and Team Makotu, have tried our best to make a 100% re-production of their submissions using all the submission files we can access, but failed. Here is just another coincidence about them (more coincidences can be seen in thread1, thread2, thread3):
Here is the heatmap of all the similarities(Notebook).
\n
\n![\"heamap.png\"](\"https://i.postimg.cc/xdtKyxtm/heatmap.png\")
\nP.S. As @shimacos suggested, all the datasets will be published soon. If anyone is interested in the challenge of recreating submission, go ahead please!
\nP.P.S. Our final submissions contain nothing from these guys. We were working separately just since the next day of team forming. Call for a precise justice.
---------------------------------------------update 2023-02-09--------------------------------
\nDatasets for anomaly detection
We, Team Bruce and Team Makotu, have tried our best to make a 100% re-production of their submissions using all the submission files we can access, but failed. Here is just another coincidence about them (more coincidences can be seen in thread1, thread2, thread3):
Here is the heatmap of all the similarities(Notebook).
\n
\n
\nP.S. As @shimacos suggested, all the datasets will be published soon. If anyone is interested in the challenge of recreating submission, go ahead please!
\nP.P.S. Our final submissions contain nothing from these guys. We were working separately just since the next day of team forming. Call for a precise justice.
---------------------------------------------update 2023-02-09--------------------------------
\nDatasets for anomaly detection
Some more data points using my team's submissions, obtained AFTER merging, i.e. we have shared ideas.
\nI compare the following submissions :
Similarities of our individual pipelines :
\nSimilarities with the ensemble :
\nObservations :
\nWhat we can infer regarind Poteman-Bruce and RunningZ-Makotu submissions:
\nPoke @inversion
\nThere is more than enough evidence that the two people in question have broken Kaggle rules.
\nDon't let this go unpunished.
I am so surprised kaggle hasn't even clarified their decision
", "rawMarkdown": "I am so surprised kaggle hasn't even clarified their decision ", "votes": 6, "replies": [ { "id": 2135439, "postDate": "2023-02-08T16:28:07.450Z", "content": "Unfortunately they don't need to.
\nAs it is written in Terms of Use Kaggle team can eliminate or not eliminate anybody only based on their own judgement or feeling.
\nNo evidence is necessary from their side and no evidence can help an unlawfully eliminated participant to get the position back if Kaggle team decides.
Maybe you can post a heatmap of all possible combinations of these submissions, which will be more easier to highlight the abnormality than just numbers.
", "rawMarkdown": "Maybe you can post a heatmap of all possible combinations of these submissions, which will be more easier to highlight the abnormality than just numbers.", "votes": 3, "replies": [ { "id": 2135203, "postDate": "2023-02-08T14:07:56.907Z", "content": "Thanks. Heatmap added
", "rawMarkdown": "Thanks. Heatmap added" } ] }, { "id": 2135285, "postDate": "2023-02-08T14:45:12.623Z", "content": "What's the public score of the runningz and poteman's sub ?
\nIf it's a bit lower, then it Would be interesting to see the cases where there is a mismatch.
\nHave they been copied from a public high scoring notebook ?
\nHas it been random replacement of aids, with a calculation to bring the score down by a fix amount. If some aids are added by random, they most probably would not increase the score at all, and x% of such aids would bring the score down by y%
", "rawMarkdown": "What's the public score of the runningz and poteman's sub ? \n\nIf it's a bit lower, then it Would be interesting to see the cases where there is a mismatch. \n\nHave they been copied from a public high scoring notebook ? \n\nHas it been random replacement of aids, with a calculation to bring the score down by a fix amount. If some aids are added by random, they most probably would not increase the score at all, and x% of such aids would bring the score down by y%", "replies": [ { "id": 2136033, "postDate": "2023-02-09T03:58:54.970Z", "content": "RunningZ's sub and Poteman's sub, used in the above analysis, scored 0.603 and 0.596 respectively. It's hard to tell whether they blended some public notebook with small weight.
", "rawMarkdown": "RunningZ's sub and Poteman's sub, used in the above analysis, scored 0.603 and 0.596 respectively. It's hard to tell whether they blended some public notebook with small weight." } ] }, { "id": 2135097, "postDate": "2023-02-08T12:59:01.850Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2135154, "author_name": "Theo Viel", "author_url": "", "post_date": "2023-02-08T13:37:34.033000", "content": "Some more data points using my team's submissions, obtained AFTER merging, i.e. we have shared ideas.
\nI compare the following submissions :
Similarities of our individual pipelines :
\nSimilarities with the ensemble :
\nObservations :
\nWhat we can infer regarind Poteman-Bruce and RunningZ-Makotu submissions:
\nPoke @inversion
\nThere is more than enough evidence that the two people in question have broken Kaggle rules.
\nDon't let this go unpunished.
I am so surprised kaggle hasn't even clarified their decision
", "votes": 6, "replies": [ { "id": 2135439, "author_name": "Allie K.", "author_url": "", "post_date": "2023-02-08T16:28:07.450000", "content": "Unfortunately they don't need to.
\nAs it is written in Terms of Use Kaggle team can eliminate or not eliminate anybody only based on their own judgement or feeling.
\nNo evidence is necessary from their side and no evidence can help an unlawfully eliminated participant to get the position back if Kaggle team decides.
Maybe you can post a heatmap of all possible combinations of these submissions, which will be more easier to highlight the abnormality than just numbers.
", "votes": 3, "replies": [ { "id": 2135203, "author_name": "sirius", "author_url": "", "post_date": "2023-02-08T14:07:56.907000", "content": "Thanks. Heatmap added
", "votes": 0, "replies": [] } ] }, { "id": 2135285, "author_name": "NikhilMishra", "author_url": "", "post_date": "2023-02-08T14:45:12.623000", "content": "What's the public score of the runningz and poteman's sub ?
\nIf it's a bit lower, then it Would be interesting to see the cases where there is a mismatch.
\nHave they been copied from a public high scoring notebook ?
\nHas it been random replacement of aids, with a calculation to bring the score down by a fix amount. If some aids are added by random, they most probably would not increase the score at all, and x% of such aids would bring the score down by y%
", "votes": 0, "replies": [ { "id": 2136033, "author_name": "sirius", "author_url": "", "post_date": "2023-02-09T03:58:54.970000", "content": "RunningZ's sub and Poteman's sub, used in the above analysis, scored 0.603 and 0.596 respectively. It's hard to tell whether they blended some public notebook with small weight.
", "votes": 0, "replies": [] } ] }, { "id": 2135097, "author_name": "", "author_url": "", "post_date": "2023-02-08T12:59:01.850000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2135082": "We, Team Bruce and Team Makotu, have tried our best to make a 100% re-production of their submissions using all the submission files we can access, but failed. Here is just another `coincidence` about them (more coincidences can be seen in [thread1](https://www.kaggle.com/competitions/otto-recommender-system/discussion/381321), [thread2](https://www.kaggle.com/competitions/otto-recommender-system/discussion/381318), [thread3](https://www.kaggle.com/competitions/otto-recommender-system/discussion/381360)): \n- **Average similarity** of submissions excepting RunningZ’s and Poteman’s: **56.1%**\n - [Notebook](https://www.kaggle.com/code/sirius81/otto-average-sim-of-irrelative-subs)\n - I collected 6 submission files from @bruceqdu, @mhyodo, @carnozhao, @alvor, @h4211819 and myself, which all scored 0.598~0.600. Then I calculated the similarity of C(6,2) pairs and averaged them.\n - In addition, the max value of them is 59.6%\n- The similarity of **RunningZ-Makotu(Poteman’s teammate)**: **73.1%**\n - [Notebook](https://www.kaggle.com/code/sirius81/otto-sub-verification)\n - It is abnormally high, even higher than the similarity of RunningZ-Bruce(RunningZ’s teammate), which is 71.3%.\n- The similarity of **Poteman-Bruce(RunningZ’s teammate)**: **74.6%**\n - [Notebook](https://www.kaggle.com/code/sirius81/otto-sub-verification-p)\n - It’s also abnormally high.\n\nHere is the heatmap of all the similarities([Notebook](https://www.kaggle.com/code/sirius81/otto-sub-verification-heatmap?scriptVersionId=118578717)).\n\n\nP.S. As @shimacos suggested, all the datasets will be published soon. If anyone is interested in the challenge of recreating submission, go ahead please! \nP.P.S. Our final submissions contain nothing from these guys. We were working separately just since the next day of team forming. Call for a precise justice.\n\n---------------------------------------------update 2023-02-09--------------------------------\n**Datasets for anomaly detection**\n1. Bruce’s sub and RunningZ’s sub: https://www.kaggle.com/datasets/ilyalos/otto-submissions-runningz\n2. Sirius’ sub: https://www.kaggle.com/datasets/sirius81/otto-sirius-sub-before-runningz-scoring-601?select=sirius_sub_meta.csv\n3. Makotu’s sub: https://www.kaggle.com/datasets/sirius81/otto-verification?select=makotu_doubtful_sub\n4. Alvor’s sub and Shimacos’ sub(only for runningZ’s 0.603): https://www.kaggle.com/datasets/sirius81/otto-sirius-sub-before-runningz-scoring-601?select=alvor_shimacos_sub\n5. Poteman's sub: https://www.kaggle.com/datasets/sirius81/otto-poteman-sub\n6. Some irrelative subs (for a baseline of similarity): https://www.kaggle.com/datasets/sirius81/otto-irrelative-subs\n", "2135154": "Some more data points using my team's submissions, obtained AFTER merging, i.e. we have shared ideas.\nI compare the following submissions :\n- One from Chris - LB 0.599\n- One from Benedikt - LB 0.599\n- One from myself - LB 0.601\n- A weighted blend of the 3 above, using weights 0.25, 0.25, 0.5 (theo) - LB 0.602\n\n**Similarities of our individual pipelines :**\n- Chris / Theo : 0.625\n- Chris / Benedikt : 0.584\n- Theo / Benedikt : 0.625\n\n**Similarities with the ensemble :**\n- Chris : 0.702\n- Benedikt : 0.703\n- Theo : 0.810\n\n**Observations :**\n- Even with sharing features and candidates, our pipelines achieve ~0.6 similarity.\n- 0.7 similarities are reached when comparing submissions with a blend that contains it at a 0.25 weight.\n- 0.8 similarities are reached when comparing submissions with a blend that contains it at a 0.5 weight.\n\n**What we can infer regarind Poteman-Bruce and RunningZ-Makotu submissions:**\n- Similarities that are higher than 0.7 indicate that the person used a very similar submission.\n- We already know that Poteman and RunningZ love blending, so we can safely assume :\n - Poteman used one of Bruce's sub in his blend with a weight of approx. 1/3\n - RunningZ used one of Makotu's sub in his blend with a weight of approx. 1/3\n- Poteman most likely sent Makotu's sub to RunningZ and RunningZ gave Bruce's to Poteman.\n\n\nPoke @inversion \n\nThere is more than enough evidence that the two people in question have broken Kaggle rules.\nDon't let this go unpunished.", "2135107": "Maybe you can post a heatmap of all possible combinations of these submissions, which will be more easier to highlight the abnormality than just numbers.", "2135285": "What's the public score of the runningz and poteman's sub ? \n\nIf it's a bit lower, then it Would be interesting to see the cases where there is a mismatch. \n\nHave they been copied from a public high scoring notebook ? \n\nHas it been random replacement of aids, with a calculation to bring the score down by a fix amount. If some aids are added by random, they most probably would not increase the score at all, and x% of such aids would bring the score down by y%", "2135097": "" } }