![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1606228%2F9e8775357c83e30221d5883addf21fe0%2F2022-10-21%2009.52.45.png?generation=1666317209276535&alt=media\")
I forked notebook from @SSKKNT(https://www.kaggle.com/code/sskknt/msci-citeseq-keras-quickstart-dropout/data), and edited for 3 version of that, but get the same result, it's weird:
\nhttps://www.kaggle.com/code/a6893676/msci-citeseq-keras-quickstart-dropout?scriptVersionId=108250114
\n(just 1 change: L2 - > L1)
https://www.kaggle.com/code/a6893676/msci-citeseq-keras-quickstart-dropout?scriptVersionId=108326451
\n(1 more layer)
https://www.kaggle.com/code/a6893676/msci-citeseq-keras-quickstart-dropout?scriptVersionId=108339491
\n(cut 3 layers)
It's incredible that all those 3 versions get exactly the same score: 0.810, even if I cut 3 layers. The same problem has happened to me in other comepetions, this is the first time I asked this question in the discussion zone, if it's inappropriate please notice me. ^^
", "messages": [ { "id": "1992190", "postDate": "10/17/2022 15:12:12", "content": "I forked notebook from @SSKKNT(https://www.kaggle.com/code/sskknt/msci-citeseq-keras-quickstart-dropout/data), and edited for 3 version of that, but get the same result, it's weird:
\nhttps://www.kaggle.com/code/a6893676/msci-citeseq-keras-quickstart-dropout?scriptVersionId=108250114
\n(just 1 change: L2 - > L1)
https://www.kaggle.com/code/a6893676/msci-citeseq-keras-quickstart-dropout?scriptVersionId=108326451
\n(1 more layer)
https://www.kaggle.com/code/a6893676/msci-citeseq-keras-quickstart-dropout?scriptVersionId=108339491
\n(cut 3 layers)
It's incredible that all those 3 versions get exactly the same score: 0.810, even if I cut 3 layers. The same problem has happened to me in other comepetions, this is the first time I asked this question in the discussion zone, if it's inappropriate please notice me. ^^
", "rawMarkdown": "I forked notebook from @SSKKNT(https://www.kaggle.com/code/sskknt/msci-citeseq-keras-quickstart-dropout/data), and edited for 3 version of that, but get the same result, it's weird:\n\nhttps://www.kaggle.com/code/a6893676/msci-citeseq-keras-quickstart-dropout?scriptVersionId=108250114\n(just 1 change: L2 - > L1)\n\nhttps://www.kaggle.com/code/a6893676/msci-citeseq-keras-quickstart-dropout?scriptVersionId=108326451\n(1 more layer)\n\nhttps://www.kaggle.com/code/a6893676/msci-citeseq-keras-quickstart-dropout?scriptVersionId=108339491\n(cut 3 layers)\n\nIt's incredible that all those 3 versions get exactly the same score: 0.810, even if I cut 3 layers. The same problem has happened to me in other comepetions, this is the first time I asked this question in the discussion zone, if it's inappropriate please notice me. ^^", "votes": null }, { "id": "1992885", "postDate": "10/18/2022 02:00:27", "content": "There are a huge number of entries with 0.810 score. Given so many individual predicted values (> 65M) and the use of average r-squared per row of data, small changes in predictions lead to very small changes in scores. The rankings of all the 0.811 and 0.810 scores is coming down to digits much further to the right of the decimal.
\nFor your results, have you looked at the correlations of the predictions you have made? I would bet that they are very highly correlated - > 0.98 or 0.99. If they are that close to each other, you won't be able to see the change in the 3 digit scores on the leaderboard.
", "rawMarkdown": "There are a huge number of entries with 0.810 score. Given so many individual predicted values (> 65M) and the use of average r-squared per row of data, small changes in predictions lead to very small changes in scores. The rankings of all the 0.811 and 0.810 scores is coming down to digits much further to the right of the decimal.\n\nFor your results, have you looked at the correlations of the predictions you have made? I would bet that they are very highly correlated - > 0.98 or 0.99. If they are that close to each other, you won't be able to see the change in the 3 digit scores on the leaderboard.", "votes": null }, { "id": "1992925", "postDate": "10/18/2022 02:48:11", "content": "Thank you much@kirkdco, I guess the correlations of the predictions should be as you bet, but for me it's incredible that I move 3 layers out of 5 layers of neural net work and got so small of a difference in result. The notebook is forked from the popular one and I think the 5 layers should count as important.
", "rawMarkdown": "Thank you much@kirkdco, I guess the correlations of the predictions should be as you bet, but for me it's incredible that I move 3 layers out of 5 layers of neural net work and got so small of a difference in result. The notebook is forked from the popular one and I think the 5 layers should count as important.", "votes": null }, { "id": "1992996", "postDate": "10/18/2022 04:07:10", "content": "I agree, that is surprising. I would be interested to see how the correlations change between predictions, though. It might be equally surprising. 8^D
", "rawMarkdown": "I agree, that is surprising. I would be interested to see how the correlations change between predictions, though. It might be equally surprising. 8^D", "votes": null }, { "id": "1993084", "postDate": "10/18/2022 05:37:23", "content": "I'm sorry I don't know how to do the correlation, but I download all 3 \"submission.csv\" results and compared them and found that they are not the same, which surprise me.
\nAt first I think they are the same, which mean kaggle has a problem. But now I don't think so.
\nThank you very much, your help make me get some progress.
Interesting. It is hard to know what is going on there.
\nRegarding correlations, if you have all the predictions as different columns in a dataframe (one column for each submission's predictions), you get a nice correlation matrix with this code:
\nimport pandas as pd
\npreds_flat = pd.DataFrame('sub1': [predictions], 'sub2': [predictions], 'sub3': [predictions])
\ndf_corr = preds_flat.corr()
\ndf_corr.style.background_gradient(cmap='coolwarm')
Thank you.
\nDo you mean that \"sub1\", \"sub2\", \"sub3\" are column title?
\nWould you please explain what should I replace \"predictions\" with?
I'm sorry for late response, there is some issue with me recently.
", "rawMarkdown": "Thank you.\nDo you mean that \"sub1\", \"sub2\", \"sub3\" are column title?\nWould you please explain what should I replace \"predictions\" with?\n\nI'm sorry for late response, there is some issue with me recently.", "votes": null }, { "id": "1996037", "postDate": "10/20/2022 03:23:44", "content": "Yes, sub1, sub2, and sub3 would be the column names. The [predictions] would be lists of the predicted values for each submission. Then, each column would be the predictions from each of your submission files. You can then get the correlations and see how's similar they are to each other.
\nNo apology necessary! I hope you're issue resolves itself and you're doing well.
", "rawMarkdown": "Yes, sub1, sub2, and sub3 would be the column names. The [predictions] would be lists of the predicted values for each submission. Then, each column would be the predictions from each of your submission files. You can then get the correlations and see how's similar they are to each other.\n\nNo apology necessary! I hope you're issue resolves itself and you're doing well.", "votes": null }, { "id": "1997577", "postDate": "10/21/2022 01:53:32", "content": "Thanks, this is the result, and later I'll adjunct my code in case the operation of my coding is wrong:
\n
This is my code:
\ntarget = pd.read_csv(\"/Users/leo/Downloads/submissionAll.csv\", index_col='row_id')
\ndf_corr = target.corr()
\ndf_corr.style.background_gradient(cmap='coolwarm')
This is the snap shot of \"submissionAll.csv\":
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1606228%2Fe782cec9d2a09d8c6e3dfc02d18bc178%2F2022-10-21%2009.56.14.png?generation=1666317395963191&alt=media\")
From the correlation matrix, it looks like your submissions are very closely related with correlation of ~0.96 between each pair of submissions. Looking at the DataFrame, though, the submissions look identical, at least for the first 25 rows. Would you expect the different submissions share the same predictions for these first rows? Since the columns are not perfectly correlated, I would anticipate that some of the later rows are different between submissions. Obviously, with 65,000,000+ rows, it is impossible to visually inspect them all. 😃
", "rawMarkdown": "From the correlation matrix, it looks like your submissions are very closely related with correlation of ~0.96 between each pair of submissions. Looking at the DataFrame, though, the submissions look identical, at least for the first 25 rows. Would you expect the different submissions share the same predictions for these first rows? Since the columns are not perfectly correlated, I would anticipate that some of the later rows are different between submissions. Obviously, with 65,000,000+ rows, it is impossible to visually inspect them all. 😃", "votes": null }, { "id": "1997609", "postDate": "10/21/2022 03:37:46", "content": "I don't expect the first rows are the same。 And the last rows are different^^.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1606228%2Fe2c41fa68a6670ea51f62ee412ad5351%2F2022-10-21%2011.37.12.png?generation=1666323460779612&alt=media\")
They way the submission file is organized, the first 6 million+ rows are predictions for the CITE data and the last 59 million+ rows are for the Multiome data. Is it possible that your submission files all use the same model for the CITE data and only have different models for the Multiome data?
\nAlso, looking the last table you posted, I would bet that despite the values in different columns having different magnitudes, they columns are reasonably well correlated. There are a number of values in the first column from -8.0 to -8.5. In the second column, those same rows have values around -2.7 to -2.8 and in the third column the values are around -7.3 to -7.5. While the magnitudes are different, the correlation may be fairly high.
", "rawMarkdown": "They way the submission file is organized, the first 6 million+ rows are predictions for the CITE data and the last 59 million+ rows are for the Multiome data. Is it possible that your submission files all use the same model for the CITE data and only have different models for the Multiome data? \n\nAlso, looking the last table you posted, I would bet that despite the values in different columns having different magnitudes, they columns are reasonably well correlated. There are a number of values in the first column from -8.0 to -8.5. In the second column, those same rows have values around -2.7 to -2.8 and in the third column the values are around -7.3 to -7.5. While the magnitudes are different, the correlation may be fairly high.", "votes": null }, { "id": "2000737", "postDate": "10/23/2022 14:09:09", "content": "I typed in all the values in the table image you posted and looked that the correlations between those columns. They all have correlations > 0.999! As I mentioned, despite the values being very different in magnitude between the columns, the correlation is still very high.
\nGiven that the first few rows are the same, I'm guessing the predictions for CITE are exactly the same for the full set of 6 million+ predictions. The Multiome preditions look like they are also very highly correlated, based on this small sample. This would explain why your different submissions have very similar scores on the leader board. And, as you mentioned before, it is very surprising given the difference in the neural networks, but it suggests to me that the smaller networks are fully capturing all the necessary information to make the same predictions as the larger datasets.
\nHere are some plots. Notice the slightly different scales on the axes. Columns 1 and 3 have a larger scale compared to column 2 - about 2X in most cases.
\nColumn 1 vs Column 2
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F779570%2F86923e5d7b0d5e9b8fef483da74c6b6a%2Fc1vc2.png?generation=1666533936178220&alt=media\")
Column 1 vs Column 3
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F779570%2F43556163eb2244acfef0921427554dad%2Fc1vc3.png?generation=1666533958870965&alt=media\")
Column 2 vs Column 3
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F779570%2Faf304a8e36d621792c4bc652ad52edc7%2Fc2vc3.png?generation=1666533997518086&alt=media\")
Hi @a6893676 , they have different scores, but the difference appears in the 4th and further decimals. In the leaderboard, we only see 3 decimals. At the end of the competition, more decimals will be revealed and you will be able to see the difference.
", "rawMarkdown": "Hi @a6893676 , they have different scores, but the difference appears in the 4th and further decimals. In the leaderboard, we only see 3 decimals. At the end of the competition, more decimals will be revealed and you will be able to see the difference.", "votes": null }, { "id": "2012813", "postDate": "11/01/2022 12:51:27", "content": "Yes, my submission files all use the same model for the CITE data and only have different models for the Multiome data. But still I think the results should be much more different.
", "rawMarkdown": "Yes, my submission files all use the same model for the CITE data and only have different models for the Multiome data. But still I think the results should be much more different.", "votes": null } ], "comments": [ { "id": 1992885, "author_name": "kirkdco", "author_url": "", "post_date": "10/18/2022 02:00:27", "content": "There are a huge number of entries with 0.810 score. Given so many individual predicted values (> 65M) and the use of average r-squared per row of data, small changes in predictions lead to very small changes in scores. The rankings of all the 0.811 and 0.810 scores is coming down to digits much further to the right of the decimal.
\nFor your results, have you looked at the correlations of the predictions you have made? I would bet that they are very highly correlated - > 0.98 or 0.99. If they are that close to each other, you won't be able to see the change in the 3 digit scores on the leaderboard.
", "votes": null, "replies": [ { "id": 1992925, "author_name": "a6893676", "author_url": "", "post_date": "10/18/2022 02:48:11", "content": "Thank you much@kirkdco, I guess the correlations of the predictions should be as you bet, but for me it's incredible that I move 3 layers out of 5 layers of neural net work and got so small of a difference in result. The notebook is forked from the popular one and I think the 5 layers should count as important.
", "votes": null, "replies": [] }, { "id": 1992996, "author_name": "kirkdco", "author_url": "", "post_date": "10/18/2022 04:07:10", "content": "I agree, that is surprising. I would be interested to see how the correlations change between predictions, though. It might be equally surprising. 8^D
", "votes": null, "replies": [] }, { "id": 1993084, "author_name": "a6893676", "author_url": "", "post_date": "10/18/2022 05:37:23", "content": "I'm sorry I don't know how to do the correlation, but I download all 3 \"submission.csv\" results and compared them and found that they are not the same, which surprise me.
\nAt first I think they are the same, which mean kaggle has a problem. But now I don't think so.
\nThank you very much, your help make me get some progress.
Interesting. It is hard to know what is going on there.
\nRegarding correlations, if you have all the predictions as different columns in a dataframe (one column for each submission's predictions), you get a nice correlation matrix with this code:
\nimport pandas as pd
\npreds_flat = pd.DataFrame('sub1': [predictions], 'sub2': [predictions], 'sub3': [predictions])
\ndf_corr = preds_flat.corr()
\ndf_corr.style.background_gradient(cmap='coolwarm')
Thank you.
\nDo you mean that \"sub1\", \"sub2\", \"sub3\" are column title?
\nWould you please explain what should I replace \"predictions\" with?
I'm sorry for late response, there is some issue with me recently.
", "votes": null, "replies": [] }, { "id": 1996037, "author_name": "kirkdco", "author_url": "", "post_date": "10/20/2022 03:23:44", "content": "Yes, sub1, sub2, and sub3 would be the column names. The [predictions] would be lists of the predicted values for each submission. Then, each column would be the predictions from each of your submission files. You can then get the correlations and see how's similar they are to each other.
\nNo apology necessary! I hope you're issue resolves itself and you're doing well.
", "votes": null, "replies": [] }, { "id": 1997577, "author_name": "a6893676", "author_url": "", "post_date": "10/21/2022 01:53:32", "content": "Thanks, this is the result, and later I'll adjunct my code in case the operation of my coding is wrong:
\n
This is my code:
\ntarget = pd.read_csv(\"/Users/leo/Downloads/submissionAll.csv\", index_col='row_id')
\ndf_corr = target.corr()
\ndf_corr.style.background_gradient(cmap='coolwarm')
This is the snap shot of \"submissionAll.csv\":
\n
From the correlation matrix, it looks like your submissions are very closely related with correlation of ~0.96 between each pair of submissions. Looking at the DataFrame, though, the submissions look identical, at least for the first 25 rows. Would you expect the different submissions share the same predictions for these first rows? Since the columns are not perfectly correlated, I would anticipate that some of the later rows are different between submissions. Obviously, with 65,000,000+ rows, it is impossible to visually inspect them all. 😃
", "votes": null, "replies": [] }, { "id": 1997609, "author_name": "a6893676", "author_url": "", "post_date": "10/21/2022 03:37:46", "content": "I don't expect the first rows are the same。 And the last rows are different^^.
\n
They way the submission file is organized, the first 6 million+ rows are predictions for the CITE data and the last 59 million+ rows are for the Multiome data. Is it possible that your submission files all use the same model for the CITE data and only have different models for the Multiome data?
\nAlso, looking the last table you posted, I would bet that despite the values in different columns having different magnitudes, they columns are reasonably well correlated. There are a number of values in the first column from -8.0 to -8.5. In the second column, those same rows have values around -2.7 to -2.8 and in the third column the values are around -7.3 to -7.5. While the magnitudes are different, the correlation may be fairly high.
", "votes": null, "replies": [] }, { "id": 2000737, "author_name": "kirkdco", "author_url": "", "post_date": "10/23/2022 14:09:09", "content": "I typed in all the values in the table image you posted and looked that the correlations between those columns. They all have correlations > 0.999! As I mentioned, despite the values being very different in magnitude between the columns, the correlation is still very high.
\nGiven that the first few rows are the same, I'm guessing the predictions for CITE are exactly the same for the full set of 6 million+ predictions. The Multiome preditions look like they are also very highly correlated, based on this small sample. This would explain why your different submissions have very similar scores on the leader board. And, as you mentioned before, it is very surprising given the difference in the neural networks, but it suggests to me that the smaller networks are fully capturing all the necessary information to make the same predictions as the larger datasets.
\nHere are some plots. Notice the slightly different scales on the axes. Columns 1 and 3 have a larger scale compared to column 2 - about 2X in most cases.
\nColumn 1 vs Column 2
\n
Column 1 vs Column 3
\n
Column 2 vs Column 3
\n
Yes, my submission files all use the same model for the CITE data and only have different models for the Multiome data. But still I think the results should be much more different.
", "votes": null, "replies": [] } ] }, { "id": 2000776, "author_name": "gehallak", "author_url": "", "post_date": "10/23/2022 14:27:57", "content": "Hi @a6893676 , they have different scores, but the difference appears in the 4th and further decimals. In the leaderboard, we only see 3 decimals. At the end of the competition, more decimals will be revealed and you will be able to see the difference.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1992190": "I forked notebook from @SSKKNT(https://www.kaggle.com/code/sskknt/msci-citeseq-keras-quickstart-dropout/data), and edited for 3 version of that, but get the same result, it's weird:\n\nhttps://www.kaggle.com/code/a6893676/msci-citeseq-keras-quickstart-dropout?scriptVersionId=108250114\n(just 1 change: L2 - > L1)\n\nhttps://www.kaggle.com/code/a6893676/msci-citeseq-keras-quickstart-dropout?scriptVersionId=108326451\n(1 more layer)\n\nhttps://www.kaggle.com/code/a6893676/msci-citeseq-keras-quickstart-dropout?scriptVersionId=108339491\n(cut 3 layers)\n\nIt's incredible that all those 3 versions get exactly the same score: 0.810, even if I cut 3 layers. The same problem has happened to me in other comepetions, this is the first time I asked this question in the discussion zone, if it's inappropriate please notice me. ^^", "1992885": "There are a huge number of entries with 0.810 score. Given so many individual predicted values (> 65M) and the use of average r-squared per row of data, small changes in predictions lead to very small changes in scores. The rankings of all the 0.811 and 0.810 scores is coming down to digits much further to the right of the decimal.\n\nFor your results, have you looked at the correlations of the predictions you have made? I would bet that they are very highly correlated - > 0.98 or 0.99. If they are that close to each other, you won't be able to see the change in the 3 digit scores on the leaderboard.", "1992925": "Thank you much@kirkdco, I guess the correlations of the predictions should be as you bet, but for me it's incredible that I move 3 layers out of 5 layers of neural net work and got so small of a difference in result. The notebook is forked from the popular one and I think the 5 layers should count as important.", "1992996": "I agree, that is surprising. I would be interested to see how the correlations change between predictions, though. It might be equally surprising. 8^D", "1993084": "I'm sorry I don't know how to do the correlation, but I download all 3 \"submission.csv\" results and compared them and found that they are not the same, which surprise me.\n\nAt first I think they are the same, which mean kaggle has a problem. But now I don't think so.\nThank you very much, your help make me get some progress.", "1994265": "Interesting. It is hard to know what is going on there.\n\nRegarding correlations, if you have all the predictions as different columns in a dataframe (one column for each submission's predictions), you get a nice correlation matrix with this code:\n\nimport pandas as pd\n\npreds_flat = pd.DataFrame('sub1': [predictions], 'sub2': [predictions], 'sub3': [predictions]) \n\ndf_corr = preds_flat.corr()\ndf_corr.style.background_gradient(cmap='coolwarm')", "1995969": "Thank you.\nDo you mean that \"sub1\", \"sub2\", \"sub3\" are column title?\nWould you please explain what should I replace \"predictions\" with?\n\nI'm sorry for late response, there is some issue with me recently.", "1996037": "Yes, sub1, sub2, and sub3 would be the column names. The [predictions] would be lists of the predicted values for each submission. Then, each column would be the predictions from each of your submission files. You can then get the correlations and see how's similar they are to each other.\n\nNo apology necessary! I hope you're issue resolves itself and you're doing well.", "1997577": "Thanks, this is the result, and later I'll adjunct my code in case the operation of my coding is wrong:\n\n", "1997578": "This is my code:\ntarget = pd.read_csv(\"/Users/leo/Downloads/submissionAll.csv\", index_col='row_id')\ndf_corr = target.corr()\ndf_corr.style.background_gradient(cmap='coolwarm')\n\nThis is the snap shot of \"submissionAll.csv\":\n", "1997580": "From the correlation matrix, it looks like your submissions are very closely related with correlation of ~0.96 between each pair of submissions. Looking at the DataFrame, though, the submissions look identical, at least for the first 25 rows. Would you expect the different submissions share the same predictions for these first rows? Since the columns are not perfectly correlated, I would anticipate that some of the later rows are different between submissions. Obviously, with 65,000,000+ rows, it is impossible to visually inspect them all. 😃", "1997609": "I don't expect the first rows are the same。 And the last rows are different^^.\n", "2000700": "They way the submission file is organized, the first 6 million+ rows are predictions for the CITE data and the last 59 million+ rows are for the Multiome data. Is it possible that your submission files all use the same model for the CITE data and only have different models for the Multiome data? \n\nAlso, looking the last table you posted, I would bet that despite the values in different columns having different magnitudes, they columns are reasonably well correlated. There are a number of values in the first column from -8.0 to -8.5. In the second column, those same rows have values around -2.7 to -2.8 and in the third column the values are around -7.3 to -7.5. While the magnitudes are different, the correlation may be fairly high.", "2000737": "I typed in all the values in the table image you posted and looked that the correlations between those columns. They all have correlations > 0.999! As I mentioned, despite the values being very different in magnitude between the columns, the correlation is still very high. \n\nGiven that the first few rows are the same, I'm guessing the predictions for CITE are exactly the same for the full set of 6 million+ predictions. The Multiome preditions look like they are also very highly correlated, based on this small sample. This would explain why your different submissions have very similar scores on the leader board. And, as you mentioned before, it is very surprising given the difference in the neural networks, but it suggests to me that the smaller networks are fully capturing all the necessary information to make the same predictions as the larger datasets.\n\nHere are some plots. Notice the slightly different scales on the axes. Columns 1 and 3 have a larger scale compared to column 2 - about 2X in most cases.\n\nColumn 1 vs Column 2\n\n\n\nColumn 1 vs Column 3\n\n \n\nColumn 2 vs Column 3\n\n", "2000776": "Hi @a6893676 , they have different scores, but the difference appears in the 4th and further decimals. In the leaderboard, we only see 3 decimals. At the end of the competition, more decimals will be revealed and you will be able to see the difference.", "2012813": "Yes, my submission files all use the same model for the CITE data and only have different models for the Multiome data. But still I think the results should be much more different." }, "source": "meta" }