![\"t-SNE](\"https://i.ibb.co/MMbDJqV/plot-t-SNE-anim-01.gif\")
This is similar to what I already discussed here. This time we have a little bit better stacking neural network, and a movie showing the whole process of how data embedding happens with t-SNE. I think it is a good approximation for how the neural network makes the final classification step. Shown at the bottom is the movie's final frame if you want to inspect it at higher resolution.
\n
![\"t-SNE](\"https://i.ibb.co/Vm9qgvG/t-SNE-keras-training-anim-01.png\")
This is similar to what I already discussed here. This time we have a little bit better stacking neural network, and a movie showing the whole process of how data embedding happens with t-SNE. I think it is a good approximation for how the neural network makes the final classification step. Shown at the bottom is the movie's final frame if you want to inspect it at higher resolution.
\n
WOW, So beautiful
", "rawMarkdown": "WOW, So beautiful", "votes": 1 }, { "id": 1898350, "postDate": "2022-08-14T14:29:32.193Z", "content": "That is so cool!
", "rawMarkdown": "That is so cool!", "votes": 1 }, { "id": 1896862, "postDate": "2022-08-13T08:19:29.353Z", "content": "that's really cool!
", "rawMarkdown": "that's really cool!", "votes": 1 }, { "id": 1895468, "postDate": "2022-08-12T06:48:43.100Z", "content": "@tilii7 Fantastic visual & frames of manifold learning.. thanks for sharing!
", "rawMarkdown": "@tilii7 Fantastic visual & frames of manifold learning.. thanks for sharing!", "votes": 1 }, { "id": 1893457, "postDate": "2022-08-10T20:37:46.253Z", "content": "That visualization is incredible, do you mind sharing a sample code, it would be interesting to see how data is being classified in other cases. Does the visualization take long to run on say a sample set of 100,000 rows with 15 features?
", "rawMarkdown": "That visualization is incredible, do you mind sharing a sample code, it would be interesting to see how data is being classified in other cases. Does the visualization take long to run on say a sample set of 100,000 rows with 15 features? ", "votes": 1, "replies": [ { "id": 1893523, "postDate": "2022-08-10T22:14:50.777Z", "content": "First thing first: this is only an approximation of how data is classified. The input data consists of penultimate layer activations (50 features), which in case of the neural network go into the final sigmoid layer where the classification is done. What t-SNE does with these activations is not the same: it will reduce data dimensionality from 50 to 2, and spread the points around the plot in such a way that immediate sample neighbors are close to each other, but their global positioning may not necessarily be representative of their true distances. It just so happens that during the t-SNE process some interesting trajectories are created, and the final result is often illustrative as well.
\nIt doesn't take long. openTSNE is multithreaded, so for this sample (~91K points x 50 features) and with 10 CPUs it takes about half an hour to do the embedding and create the animation. The input data is obtained by cutting out the last neural network layer (the sigmoid layer) and doing predictions, which generates a 50-vector prediction for each data point since the penultimate layer has 50 neurons. You can choose a smaller number of neurons in that layer, so the embedding could be even faster. Once the input data is available, openTSNE takes over and it is pretty straightforward. I use perplexity of 40-100 (100 for the above animation) and there is an explanation on their website how to create an animation of the whole embedding. I typically cut out the early exaggeration phase (also explained) because not much happens visually during that time.
", "rawMarkdown": "First thing first: this is only an approximation of how data is classified. The input data consists of penultimate layer activations (50 features), which in case of the neural network go into the final sigmoid layer where the classification is done. What t-SNE does with these activations is not the same: it will reduce data dimensionality from 50 to 2, and spread the points around the plot in such a way that immediate sample neighbors are close to each other, but their global positioning may not necessarily be representative of their true distances. It just so happens that during the t-SNE process some interesting trajectories are created, and the final result is often illustrative as well.\n\nIt doesn't take long. [**openTSNE**](https://github.com/pavlin-policar/openTSNE) is multithreaded, so for this sample (~91K points x 50 features) and with 10 CPUs it takes about half an hour to do the embedding and create the animation. The input data is obtained by cutting out the last neural network layer (the sigmoid layer) and doing predictions, which generates a 50-vector prediction for each data point since the penultimate layer has 50 neurons. You can choose a smaller number of neurons in that layer, so the embedding could be even faster. Once the input data is available, openTSNE takes over and it is pretty straightforward. I use perplexity of 40-100 (100 for the above animation) and there is an explanation on their website how to create an animation of the whole embedding. I typically cut out the early exaggeration phase (also explained) because not much happens visually during that time.", "votes": 3 }, { "id": 1899656, "postDate": "2022-08-15T12:31:32.207Z", "content": "Do you have an example how to use open tsne ?
", "rawMarkdown": "Do you have an example how to use open tsne ?" }, { "id": 1899869, "postDate": "2022-08-15T14:58:24.310Z", "content": "I don't have any code I could share, because most of my t-SNE code is written for other purposes. It is very simple to get it going, and they have extensive documentation. Any kind of tabular data works (class/target values should be excluded), and it is a good idea to normalize the values. For <50,000 data points, what I show below should work fine. For >50,000 data points, I suggest changing early_exaggeration_iter=2000 and n_iter=5000. You may want to experiment with perplexity values for any specific application, though for DNA embedding from 4n frequencies 20-40 works fine. Also, openTSNE will happily use all the CPUs you have, so you may want to increase n_jobs depending on your system.
from openTSNE import TSNE\n\n# load tabular data into train_data, normalize if needed\n\ntsne = TSNE(\n perplexity=40,\n metric='cosine',\n n_jobs=10,\n learning_rate='auto',\n negative_gradient_method='bh',\n theta=0.5,\n initialization='spectral',\n early_exaggeration_iter=1000,\n early_exaggeration=18.0,\n n_iter=3000,\n initial_momentum=0.5,\n final_momentum=0.8,\n verbose=1)\n\nembedding = tsne.fit(train_data)\nThis is really interesting, check my new article on Devgenius classifying bacteria DNA segments using tsne :
\nhttps://medium.com/dev-genius/data-science-exploratory-analysis-code-blocks-used-by-data-engineers-ce4529b70672
Nicely done. You should know that sklearn's t-SNE implementation is really substandard for these kinds of jobs, and I think in general.
\nHundreds of microbial genomes can be separated from a complex mixture using openTSNE. The highest cluster number in the image below is 351, and that's before the refinment.
\n![\"t-SNE](\"https://i.ibb.co/QXQ1jvC/contigs-hdbscan-20k-5k-2-D-opent-SNE-clusters.png\")
Do you know how can I extract the segments dataset after analysis ?
", "rawMarkdown": "Do you know how can I extract the segments dataset after analysis ?" }, { "id": 1893592, "postDate": "2022-08-10T23:27:46.257Z", "content": "I use scripts from the CONCOCT package, but that assumes cutting up DNA into pieces the way they do it. Relevant scripts are extract_fasta_bins.py and merge_cutup_clustering.py. You may want to go end-to-end with CONCOCT, though I get better results for complex datasets with openTSNE.
Indeed very interesting I will advise you to write an article on this 👍👍
", "rawMarkdown": "Indeed very interesting I will advise you to write an article on this 👍👍", "votes": 1 }, { "id": 1911904, "postDate": "2022-08-24T11:35:57.090Z", "content": "Do you have a code example using concoct ? thanks for your help it is just for learning new methods 😃
", "rawMarkdown": "Do you have a code example using concoct ? thanks for your help it is just for learning new methods 😃" } ] }, { "id": 1889801, "postDate": "2022-08-08T11:29:04.293Z", "content": "so cool !!
", "rawMarkdown": "so cool !!", "votes": 1 }, { "id": 1892020, "postDate": "2022-08-09T19:50:24.703Z", "content": "The animation is really cool!
", "rawMarkdown": "The animation is really cool! \n\n\n", "votes": 2 }, { "id": 1904167, "postDate": "2022-08-18T02:37:59.043Z", "content": "This is too cool
", "rawMarkdown": "This is too cool " }, { "id": 1892884, "postDate": "2022-08-10T11:59:25.670Z", "content": "Thank you for the great answer I will try the openTsne, it seems a great way to find new antibiotics even for monkeypox.
", "rawMarkdown": "Thank you for the great answer I will try the openTsne, it seems a great way to find new antibiotics even for monkeypox." }, { "id": 1900433, "postDate": "2022-08-16T03:18:50.413Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 1903290, "author_name": "", "author_url": "", "post_date": "2022-08-17T09:22:39.040000", "content": "WOW, So beautiful
", "votes": 1, "replies": [] }, { "id": 1898350, "author_name": "", "author_url": "", "post_date": "2022-08-14T14:29:32.193000", "content": "That is so cool!
", "votes": 1, "replies": [] }, { "id": 1896862, "author_name": "", "author_url": "", "post_date": "2022-08-13T08:19:29.353000", "content": "that's really cool!
", "votes": 1, "replies": [] }, { "id": 1895468, "author_name": "Ranja Sarkar (She/Her)", "author_url": "", "post_date": "2022-08-12T06:48:43.100000", "content": "@tilii7 Fantastic visual & frames of manifold learning.. thanks for sharing!
", "votes": 1, "replies": [] }, { "id": 1893457, "author_name": "JP", "author_url": "", "post_date": "2022-08-10T20:37:46.253000", "content": "That visualization is incredible, do you mind sharing a sample code, it would be interesting to see how data is being classified in other cases. Does the visualization take long to run on say a sample set of 100,000 rows with 15 features?
", "votes": 1, "replies": [ { "id": 1893523, "author_name": "Tilii", "author_url": "", "post_date": "2022-08-10T22:14:50.777000", "content": "First thing first: this is only an approximation of how data is classified. The input data consists of penultimate layer activations (50 features), which in case of the neural network go into the final sigmoid layer where the classification is done. What t-SNE does with these activations is not the same: it will reduce data dimensionality from 50 to 2, and spread the points around the plot in such a way that immediate sample neighbors are close to each other, but their global positioning may not necessarily be representative of their true distances. It just so happens that during the t-SNE process some interesting trajectories are created, and the final result is often illustrative as well.
\nIt doesn't take long. openTSNE is multithreaded, so for this sample (~91K points x 50 features) and with 10 CPUs it takes about half an hour to do the embedding and create the animation. The input data is obtained by cutting out the last neural network layer (the sigmoid layer) and doing predictions, which generates a 50-vector prediction for each data point since the penultimate layer has 50 neurons. You can choose a smaller number of neurons in that layer, so the embedding could be even faster. Once the input data is available, openTSNE takes over and it is pretty straightforward. I use perplexity of 40-100 (100 for the above animation) and there is an explanation on their website how to create an animation of the whole embedding. I typically cut out the early exaggeration phase (also explained) because not much happens visually during that time.
", "votes": 3, "replies": [] }, { "id": 1899656, "author_name": "philBoaz", "author_url": "", "post_date": "2022-08-15T12:31:32.207000", "content": "Do you have an example how to use open tsne ?
", "votes": 0, "replies": [] }, { "id": 1899869, "author_name": "Tilii", "author_url": "", "post_date": "2022-08-15T14:58:24.310000", "content": "I don't have any code I could share, because most of my t-SNE code is written for other purposes. It is very simple to get it going, and they have extensive documentation. Any kind of tabular data works (class/target values should be excluded), and it is a good idea to normalize the values. For <50,000 data points, what I show below should work fine. For >50,000 data points, I suggest changing early_exaggeration_iter=2000 and n_iter=5000. You may want to experiment with perplexity values for any specific application, though for DNA embedding from 4n frequencies 20-40 works fine. Also, openTSNE will happily use all the CPUs you have, so you may want to increase n_jobs depending on your system.
from openTSNE import TSNE\n\n# load tabular data into train_data, normalize if needed\n\ntsne = TSNE(\n perplexity=40,\n metric='cosine',\n n_jobs=10,\n learning_rate='auto',\n negative_gradient_method='bh',\n theta=0.5,\n initialization='spectral',\n early_exaggeration_iter=1000,\n early_exaggeration=18.0,\n n_iter=3000,\n initial_momentum=0.5,\n final_momentum=0.8,\n verbose=1)\n\nembedding = tsne.fit(train_data)\nThis is really interesting, check my new article on Devgenius classifying bacteria DNA segments using tsne :
\nhttps://medium.com/dev-genius/data-science-exploratory-analysis-code-blocks-used-by-data-engineers-ce4529b70672
Nicely done. You should know that sklearn's t-SNE implementation is really substandard for these kinds of jobs, and I think in general.
\nHundreds of microbial genomes can be separated from a complex mixture using openTSNE. The highest cluster number in the image below is 351, and that's before the refinment.
\n
Do you know how can I extract the segments dataset after analysis ?
", "votes": 0, "replies": [] }, { "id": 1893592, "author_name": "Tilii", "author_url": "", "post_date": "2022-08-10T23:27:46.257000", "content": "I use scripts from the CONCOCT package, but that assumes cutting up DNA into pieces the way they do it. Relevant scripts are extract_fasta_bins.py and merge_cutup_clustering.py. You may want to go end-to-end with CONCOCT, though I get better results for complex datasets with openTSNE.
Indeed very interesting I will advise you to write an article on this 👍👍
", "votes": 1, "replies": [] }, { "id": 1911904, "author_name": "philBoaz", "author_url": "", "post_date": "2022-08-24T11:35:57.090000", "content": "Do you have a code example using concoct ? thanks for your help it is just for learning new methods 😃
", "votes": 0, "replies": [] } ] }, { "id": 1889801, "author_name": "fengfeng", "author_url": "", "post_date": "2022-08-08T11:29:04.293000", "content": "so cool !!
", "votes": 1, "replies": [] }, { "id": 1892020, "author_name": "The Devastator", "author_url": "", "post_date": "2022-08-09T19:50:24.703000", "content": "The animation is really cool!
", "votes": 2, "replies": [] }, { "id": 1904167, "author_name": "Pascal Yamlome", "author_url": "", "post_date": "2022-08-18T02:37:59.043000", "content": "This is too cool
", "votes": 0, "replies": [] }, { "id": 1892884, "author_name": "philBoaz", "author_url": "", "post_date": "2022-08-10T11:59:25.670000", "content": "Thank you for the great answer I will try the openTsne, it seems a great way to find new antibiotics even for monkeypox.
", "votes": 0, "replies": [] }, { "id": 1900433, "author_name": "", "author_url": "", "post_date": "2022-08-16T03:18:50.413000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1889671": "This is similar to what I already discussed [**here**](https://www.kaggle.com/competitions/amex-default-prediction/discussion/339726). This time we have a little bit better stacking neural network, and a movie showing the whole process of how data embedding happens with t-SNE. I think it is a good approximation for how the neural network makes the final classification step. Shown at the bottom is the movie's final frame if you want to inspect it at higher resolution.\n\n\n\n\n", "1903290": "WOW, So beautiful", "1898350": "That is so cool!", "1896862": "that's really cool!", "1895468": "@tilii7 Fantastic visual & frames of manifold learning.. thanks for sharing!", "1893457": "That visualization is incredible, do you mind sharing a sample code, it would be interesting to see how data is being classified in other cases. Does the visualization take long to run on say a sample set of 100,000 rows with 15 features? ", "1892175": "This is really interesting, check my new article on Devgenius classifying bacteria DNA segments using tsne : \nhttps://medium.com/dev-genius/data-science-exploratory-analysis-code-blocks-used-by-data-engineers-ce4529b70672", "1889801": "so cool !!", "1892020": "The animation is really cool! \n\n\n", "1904167": "This is too cool ", "1892884": "Thank you for the great answer I will try the openTsne, it seems a great way to find new antibiotics even for monkeypox.", "1900433": "" } }