![\"t-SNE](\"https://i.ibb.co/9rGwK67/t-SNE-keras-training-06.png\")
This is a finishing touch on a topic that started in this post. In the original post we had a single neural network model scoring 0.790 on public LB. Here we have 12 different models (a mix of NN, LGB and logistic regression; scores are 0.78-0.795 on public LB) that are stacked together by another neural network. The model scores 0.799 on public LB, and t-SNE of its activations shows much better separation between the two classes. Notice that both tips of the worm have increased density of points, as a stack of models predicts many samples with extreme certainty.
\n
EDIT: There is a plot here showing how PCA interprets this dataset.
", "messages": [ { "id": 1871311, "postDate": "2022-07-26T07:55:43.270Z", "content": "This is a finishing touch on a topic that started in this post. In the original post we had a single neural network model scoring 0.790 on public LB. Here we have 12 different models (a mix of NN, LGB and logistic regression; scores are 0.78-0.795 on public LB) that are stacked together by another neural network. The model scores 0.799 on public LB, and t-SNE of its activations shows much better separation between the two classes. Notice that both tips of the worm have increased density of points, as a stack of models predicts many samples with extreme certainty.
\n
EDIT: There is a plot here showing how PCA interprets this dataset.
", "rawMarkdown": "This is a finishing touch on a topic that started in [**this post**](https://www.kaggle.com/competitions/amex-default-prediction/discussion/339278). In the original post we had a single neural network model scoring 0.790 on public LB. Here we have 12 different models (a mix of NN, LGB and logistic regression; scores are 0.78-0.795 on public LB) that are stacked together by another neural network. The model scores 0.799 on public LB, and t-SNE of its activations shows much better separation between the two classes. Notice that both tips of the worm have increased density of points, as a stack of models predicts many samples with extreme certainty.\n\n\n\n**EDIT**: There is a plot [**here**](https://www.kaggle.com/competitions/amex-default-prediction/discussion/339278#1872271 ) showing how PCA interprets this dataset.", "votes": 50 }, { "id": 1874671, "postDate": "2022-07-28T12:22:16.187Z", "content": "I'm new to data science and understand almost nothing of what you're saying, but these visuals are lovely, in all their variations.
\nThank you for the inspiration.
There are 3 related posts I wrote on this subject, and they are cross-linked. It may help if you read them all, and feel free to ask if something specific is still unclear.
", "rawMarkdown": "There are 3 related posts I wrote on this subject, and they are cross-linked. It may help if you read them all, and feel free to ask if something specific is still unclear." }, { "id": 1875303, "postDate": "2022-07-28T22:01:18.033Z", "content": "I'll do that. Thank you.
", "rawMarkdown": "I'll do that. Thank you." } ] }, { "id": 1873136, "postDate": "2022-07-27T13:07:06.230Z", "content": "I did the same exercise and got few strange paintings :)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2596066%2Fa78dafbd0d96e43403ca7ee6ba25468e%2Ftsne_fig1.png?generation=1658926473095519&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2596066%2F838fe24580cd0bdff243481e10c85a2a%2Fumap_fig1.png?generation=1658926566640056&alt=media\")
Not sure why the t-SNE look so different from Tillli's plot and how to interpret this, but is really fun to play with it.
\nPS: I used just one fold (~91k customers) and default params for t-SNE, UMAP
\nPS: CV 0.798 / LB 0.797+
t-SNE is very sensitive to it's hyperparameters. Here is a great article I found some time ago:
\nhttps://distill.pub/2016/misread-tsne/
What @thomasmeiner said is true, except that I wouldn't use a plural. In my experience, the only parameter that truly matters is perplexity. Smaller values of perplexity tend to make plots where the dots are diffuse and form squiggles, kind of like in your plot. My original plot (not from this page, but here) was done with perplexity=20. If I use the same dataset but set perplexity=5, this is what comes out as a result.
![\"t-SNE](\"https://i.ibb.co/ZJDQZtN/t-SNE-keras-training-02a.png\")
So I suggest you try the same t-SNE implementation I referenced in the original post, and play with perplexity until you get the embedding that makes most sense. Or just leave it as is, because this is still very nice graphics π
", "rawMarkdown": "What @thomasmeiner said is true, except that I wouldn't use a plural. In my experience, the only parameter that truly matters is perplexity. Smaller values of perplexity tend to make plots where the dots are diffuse and form squiggles, kind of like in your plot. My original plot (not from this page, but [**here**](https://www.kaggle.com/competitions/amex-default-prediction/discussion/339278)) was done with `perplexity=20`. If I use the same dataset but set `perplexity=5`, this is what comes out as a result.\n\n\n\nSo I suggest you try the same t-SNE implementation I referenced in the original post, and play with perplexity until you get the embedding that makes most sense. Or just leave it as is, because this is still very nice graphics π", "votes": 2 }, { "id": 1873707, "postDate": "2022-07-27T20:33:57.987Z", "content": "yes exactly, perplexity is the key param. I played with some values and graphs change a lot.
\nFor reference my plot above is with RAPIDS cuml t-SNE(perplexity=30) which is the default value from what I see.
Don't know about RAPIDS implementation, but my experience is that sklearn's t-SNE is inferior both in terms of speed and results. Again, I recommend openTSNE.
", "rawMarkdown": "Don't know about RAPIDS implementation, but my experience is that sklearn's t-SNE is inferior both in terms of speed and results. Again, I recommend [**openTSNE**](https://github.com/pavlin-policar/openTSNE).", "votes": 1 }, { "id": 1873723, "postDate": "2022-07-27T21:04:11.190Z", "content": "Yes it is perplexity. The article linked above shows that as well.
", "rawMarkdown": "Yes it is perplexity. The article linked above shows that as well.", "votes": 1 } ] }, { "id": 1871340, "postDate": "2022-07-26T08:11:11.627Z", "content": "Dear @tilii7
\nVery nice!
\nBTW, I have found that using different colour maps can also create some pretty t-SNE plotsβ¦
All the best,
\ncarl
Since you want more color, here is the same image as above but colored according to network predictions rather than actual class values. Even though t-SNE never sees the actual class labels or neural network predictions, its embedding matches the distribution of predictions.
\n![\"t-SNE](\"https://i.ibb.co/F4Wnh1k/t-SNE-keras-training-08a.png\")
Wow, is very very cool .
", "rawMarkdown": "Wow, is very very cool .", "votes": 1 }, { "id": 1883764, "postDate": "2022-08-04T04:57:59.173Z", "content": "Wow, this is beautiful
", "rawMarkdown": "Wow, this is beautiful", "votes": 1 }, { "id": 1882853, "postDate": "2022-08-03T13:01:29.997Z", "content": "This is art! I can't think of any other words to compliment you.π
", "rawMarkdown": "This is art! I can't think of any other words to compliment you.π", "votes": 1 }, { "id": 1882197, "postDate": "2022-08-03T05:57:23.510Z", "content": "Great plot! I have to improve my skills in neural network
", "rawMarkdown": "Great plot! I have to improve my skills in neural network", "votes": 1 }, { "id": 1882071, "postDate": "2022-08-03T04:07:50.680Z", "content": "wow, great job
", "rawMarkdown": "wow, great job", "votes": 1 }, { "id": 1875807, "postDate": "2022-07-29T10:01:22.173Z", "content": "data is art indeed! +2
", "rawMarkdown": "data is art indeed! +2", "votes": 1 }, { "id": 1873819, "postDate": "2022-07-27T23:20:02.877Z", "content": "Nice one π!
\nIt is really interesting.. why this happen?
I am not sure what exactly you are asking.
\nIf you are asking why data classes separate, it is because the underlying neural network has already learned the pattern that makes them different. All that is left is for this vector of 50 numbers to be fed into a sigmoid layer, which will convert the information into sigmoidally-distributed probabilities. Instead, we feed the information into t-SNE, which will reduce the dimensionality from 50 to 2, but still maintain nice data separation.
\nIf you are asking why the overall plot looks like a worm, it is kind of explained in this comment. Briefly, t-SNE tries to keep locally similar samples together, without necessarily maintaining global distance between data points. Since the perplexity parameter was set at 20, in plain terms it has to make sure that each point has at least another 20 points nearby. That allows for quite a big stretch of data points as they don't have to be densely packed, and eventually the whole thing winds around like a worm. Changing the perplexity parameter to a large value, say 1000, will now require that each point has another 1000 points nearby. That will make the plot more compact because data points can't be spread so much, and that will likely remove the \"worminess.\" See below the same data as above in the main plot, but with perplexity=1000. Note that the scale in this plot is [-60, 60] while it is [-300, 300] above, which means that data spread is much more compact.
![\"t-SNE](\"https://i.ibb.co/8mqQGPD/t-SNE-keras-training-09.png\")
Incredibly interesting, thank you!
", "rawMarkdown": "Incredibly interesting, thank you! ~~Have you seen the worm on a string memes by chance?~~", "votes": 1 }, { "id": 1896845, "postDate": "2022-08-13T08:02:36.237Z", "content": "wow, Great job
", "rawMarkdown": "wow, Great job" }, { "id": 1875838, "postDate": "2022-07-29T10:37:02.380Z", "content": "Wow, the picture is very very cool .
\nData is Art, and BTW, my startup is named BoolArt, lol .
", "rawMarkdown": "Wow, the picture is very very cool .\n\nData is Art, and BTW, my startup is named BoolArt, lol . ", "votes": 2 }, { "id": 1874502, "postDate": "2022-07-28T10:12:14.400Z", "content": "Data is creative !
", "rawMarkdown": "Data is creative !", "votes": 2 }, { "id": 1872460, "postDate": "2022-07-27T02:01:12.110Z", "content": "beautiful plot @tilii7
", "rawMarkdown": "beautiful plot @tilii7 ", "votes": 2 }, { "id": 1872299, "postDate": "2022-07-26T19:41:42.447Z", "content": "very artistic data
", "rawMarkdown": "very artistic data", "votes": 2 }, { "id": 1871659, "postDate": "2022-07-26T11:56:44.333Z", "content": "data is art indeed! +1
", "rawMarkdown": "data is art indeed! +1", "votes": 2 }, { "id": 1884086, "postDate": "2022-08-04T09:24:30.097Z", "content": "wow, great job
", "rawMarkdown": "wow, great job", "votes": 1, "isDeleted": true }, { "id": 1883683, "postDate": "2022-08-04T02:47:29.267Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1873431, "postDate": "2022-07-27T15:52:38.490Z", "content": "thanks sir @tilii7
", "rawMarkdown": "thanks sir @tilii7 ", "votes": 1 } ], "comments": [ { "id": 1874671, "author_name": "SKLasky", "author_url": "", "post_date": "2022-07-28T12:22:16.187000", "content": "I'm new to data science and understand almost nothing of what you're saying, but these visuals are lovely, in all their variations.
\nThank you for the inspiration.
There are 3 related posts I wrote on this subject, and they are cross-linked. It may help if you read them all, and feel free to ask if something specific is still unclear.
", "votes": 0, "replies": [] }, { "id": 1875303, "author_name": "SKLasky", "author_url": "", "post_date": "2022-07-28T22:01:18.033000", "content": "I'll do that. Thank you.
", "votes": 0, "replies": [] } ] }, { "id": 1873136, "author_name": "Ioannis M", "author_url": "", "post_date": "2022-07-27T13:07:06.230000", "content": "I did the same exercise and got few strange paintings :)
\n
Not sure why the t-SNE look so different from Tillli's plot and how to interpret this, but is really fun to play with it.
\nPS: I used just one fold (~91k customers) and default params for t-SNE, UMAP
\nPS: CV 0.798 / LB 0.797+
t-SNE is very sensitive to it's hyperparameters. Here is a great article I found some time ago:
\nhttps://distill.pub/2016/misread-tsne/
What @thomasmeiner said is true, except that I wouldn't use a plural. In my experience, the only parameter that truly matters is perplexity. Smaller values of perplexity tend to make plots where the dots are diffuse and form squiggles, kind of like in your plot. My original plot (not from this page, but here) was done with perplexity=20. If I use the same dataset but set perplexity=5, this is what comes out as a result.
So I suggest you try the same t-SNE implementation I referenced in the original post, and play with perplexity until you get the embedding that makes most sense. Or just leave it as is, because this is still very nice graphics π
", "votes": 2, "replies": [] }, { "id": 1873707, "author_name": "Ioannis M", "author_url": "", "post_date": "2022-07-27T20:33:57.987000", "content": "yes exactly, perplexity is the key param. I played with some values and graphs change a lot.
\nFor reference my plot above is with RAPIDS cuml t-SNE(perplexity=30) which is the default value from what I see.
Don't know about RAPIDS implementation, but my experience is that sklearn's t-SNE is inferior both in terms of speed and results. Again, I recommend openTSNE.
", "votes": 1, "replies": [] }, { "id": 1873723, "author_name": "Thomas MeiΓner", "author_url": "", "post_date": "2022-07-27T21:04:11.190000", "content": "Yes it is perplexity. The article linked above shows that as well.
", "votes": 1, "replies": [] } ] }, { "id": 1871340, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2022-07-26T08:11:11.627000", "content": "Dear @tilii7
\nVery nice!
\nBTW, I have found that using different colour maps can also create some pretty t-SNE plotsβ¦
All the best,
\ncarl
Since you want more color, here is the same image as above but colored according to network predictions rather than actual class values. Even though t-SNE never sees the actual class labels or neural network predictions, its embedding matches the distribution of predictions.
\n
Wow, is very very cool .
", "votes": 1, "replies": [] }, { "id": 1883764, "author_name": "Will", "author_url": "", "post_date": "2022-08-04T04:57:59.173000", "content": "Wow, this is beautiful
", "votes": 1, "replies": [] }, { "id": 1882853, "author_name": "", "author_url": "", "post_date": "2022-08-03T13:01:29.997000", "content": "This is art! I can't think of any other words to compliment you.π
", "votes": 1, "replies": [] }, { "id": 1882197, "author_name": "Agustin222", "author_url": "", "post_date": "2022-08-03T05:57:23.510000", "content": "Great plot! I have to improve my skills in neural network
", "votes": 1, "replies": [] }, { "id": 1882071, "author_name": "", "author_url": "", "post_date": "2022-08-03T04:07:50.680000", "content": "wow, great job
", "votes": 1, "replies": [] }, { "id": 1875807, "author_name": "Kuzma Peng", "author_url": "", "post_date": "2022-07-29T10:01:22.173000", "content": "data is art indeed! +2
", "votes": 1, "replies": [] }, { "id": 1873819, "author_name": "The Devastator", "author_url": "", "post_date": "2022-07-27T23:20:02.877000", "content": "Nice one π!
\nIt is really interesting.. why this happen?
I am not sure what exactly you are asking.
\nIf you are asking why data classes separate, it is because the underlying neural network has already learned the pattern that makes them different. All that is left is for this vector of 50 numbers to be fed into a sigmoid layer, which will convert the information into sigmoidally-distributed probabilities. Instead, we feed the information into t-SNE, which will reduce the dimensionality from 50 to 2, but still maintain nice data separation.
\nIf you are asking why the overall plot looks like a worm, it is kind of explained in this comment. Briefly, t-SNE tries to keep locally similar samples together, without necessarily maintaining global distance between data points. Since the perplexity parameter was set at 20, in plain terms it has to make sure that each point has at least another 20 points nearby. That allows for quite a big stretch of data points as they don't have to be densely packed, and eventually the whole thing winds around like a worm. Changing the perplexity parameter to a large value, say 1000, will now require that each point has another 1000 points nearby. That will make the plot more compact because data points can't be spread so much, and that will likely remove the \"worminess.\" See below the same data as above in the main plot, but with perplexity=1000. Note that the scale in this plot is [-60, 60] while it is [-300, 300] above, which means that data spread is much more compact.
Incredibly interesting, thank you!
", "votes": 1, "replies": [] }, { "id": 1896845, "author_name": "", "author_url": "", "post_date": "2022-08-13T08:02:36.237000", "content": "wow, Great job
", "votes": 0, "replies": [] }, { "id": 1875838, "author_name": "KKY", "author_url": "", "post_date": "2022-07-29T10:37:02.380000", "content": "Wow, the picture is very very cool .
\nData is Art, and BTW, my startup is named BoolArt, lol .
", "votes": 2, "replies": [] }, { "id": 1874502, "author_name": "Shrijayan", "author_url": "", "post_date": "2022-07-28T10:12:14.400000", "content": "Data is creative !
", "votes": 2, "replies": [] }, { "id": 1872460, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-07-27T02:01:12.110000", "content": "beautiful plot @tilii7
", "votes": 2, "replies": [] }, { "id": 1872299, "author_name": "Orkhan Suleymanli", "author_url": "", "post_date": "2022-07-26T19:41:42.447000", "content": "very artistic data
", "votes": 2, "replies": [] }, { "id": 1871659, "author_name": "Ioannis M", "author_url": "", "post_date": "2022-07-26T11:56:44.333000", "content": "data is art indeed! +1
", "votes": 2, "replies": [] }, { "id": 1884086, "author_name": "", "author_url": "", "post_date": "2022-08-04T09:24:30.097000", "content": "wow, great job
", "votes": 1, "replies": [] }, { "id": 1883683, "author_name": "", "author_url": "", "post_date": "2022-08-04T02:47:29.267000", "content": "", "votes": 1, "replies": [] }, { "id": 1873431, "author_name": "Bilal Suppal", "author_url": "", "post_date": "2022-07-27T15:52:38.490000", "content": "thanks sir @tilii7
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1871311": "This is a finishing touch on a topic that started in [**this post**](https://www.kaggle.com/competitions/amex-default-prediction/discussion/339278). In the original post we had a single neural network model scoring 0.790 on public LB. Here we have 12 different models (a mix of NN, LGB and logistic regression; scores are 0.78-0.795 on public LB) that are stacked together by another neural network. The model scores 0.799 on public LB, and t-SNE of its activations shows much better separation between the two classes. Notice that both tips of the worm have increased density of points, as a stack of models predicts many samples with extreme certainty.\n\n\n\n**EDIT**: There is a plot [**here**](https://www.kaggle.com/competitions/amex-default-prediction/discussion/339278#1872271 ) showing how PCA interprets this dataset.", "1874671": "I'm new to data science and understand almost nothing of what you're saying, but these visuals are lovely, in all their variations. \nThank you for the inspiration.", "1873136": "I did the same exercise and got few strange paintings :) \n\n\n\n\n\n\n\n\n\n\nNot sure why the t-SNE look so different from Tillli's plot and how to interpret this, but is really fun to play with it. \n\nPS: I used just one fold (~91k customers) and default params for t-SNE, UMAP\nPS: CV 0.798 / LB 0.797+\n", "1871340": "Dear @tilii7 \n\nVery nice! \nBTW, I have found that using different colour maps can also create [some pretty t-SNE plots](https://www.kaggle.com/code/carlmcbrideellis/some-pretty-t-sne-plots)...\n\nAll the best,\ncarl", "1897397": "Wow, is very very cool .", "1883764": "Wow, this is beautiful", "1882853": "This is art! I can't think of any other words to compliment you.π", "1882197": "Great plot! I have to improve my skills in neural network", "1882071": "wow, great job", "1875807": "data is art indeed! +2", "1873819": "Nice one π!\nIt is really interesting.. why this happen?\n", "1873181": "Incredibly interesting, thank you! ~~Have you seen the worm on a string memes by chance?~~", "1896845": "wow, Great job", "1875838": "Wow, the picture is very very cool .\n\nData is Art, and BTW, my startup is named BoolArt, lol . ", "1874502": "Data is creative !", "1872460": "beautiful plot @tilii7 ", "1872299": "very artistic data", "1871659": "data is art indeed! +1", "1884086": "wow, great job", "1883683": "", "1873431": "thanks sir @tilii7 " } }