{ "id": 304961, "title": "Data Distribution", "url": "/competitions/happy-whale-and-dolphin/discussion/304961", "author_name": "", "post_date": "2022-02-03T06:15:53.677082200Z", "votes": 43, "comment_count": 8, "views": 0, "content": "

One of the key challenges of this competition is that we've to identify new_indiviual even though they don't appear on the train data. My initial thought was that new_individual may look very different as they don't appear on the train data. But image embeddings tell a different story. I've published a notebook to demonstrate this analysis using T-SNE.

\n

Notebook Link:

\n\n

Data Distribution:

\n

To Plot image embeddings I've used using both WandB and cuML. Both of them are great libraries. Here are some of the visualizations,

\n

Train Vs Test:

\n
\"results-53-0\"
\n

Whales Vs Dolphin:

\n
\"results-57-0\"
\n

Whales Species:

\n
\"results-61-0\"
\n

Dolphin Species:

\n
\"results-63-0\"
\n

Random Train/Test:

\n

\"trainvtest\"

\n

Random Whale/Dolphin:

\n

\"wvsd\"

\n

UMAP/PCA/TSNE using WandB

\n

We can also plot embeddings using wandb. Here's some cool plots using wandb,
\n\"w-b01\"
\n\"w-b02\"
\n\"w-b04\"

", "messages": [ { "id": "1673977", "postDate": "02/03/2022 06:15:53", "content": "

One of the key challenges of this competition is that we've to identify new_indiviual even though they don't appear on the train data. My initial thought was that new_individual may look very different as they don't appear on the train data. But image embeddings tell a different story. I've published a notebook to demonstrate this analysis using T-SNE.

\n

Notebook Link:

\n\n

Data Distribution:

\n

To Plot image embeddings I've used using both WandB and cuML. Both of them are great libraries. Here are some of the visualizations,

\n

Train Vs Test:

\n
\"results-53-0\"
\n

Whales Vs Dolphin:

\n
\"results-57-0\"
\n

Whales Species:

\n
\"results-61-0\"
\n

Dolphin Species:

\n
\"results-63-0\"
\n

Random Train/Test:

\n

\"trainvtest\"

\n

Random Whale/Dolphin:

\n

\"wvsd\"

\n

UMAP/PCA/TSNE using WandB

\n

We can also plot embeddings using wandb. Here's some cool plots using wandb,
\n\"w-b01\"
\n\"w-b02\"
\n\"w-b04\"

", "rawMarkdown": "One of the key challenges of this competition is that we've to identify `new_indiviual` even though they don't appear on the **train** data. My initial thought was that `new_individual` may look very different as they don't appear on the **train** data. But **image embeddings** tell a different story. I've published a notebook to demonstrate this analysis using **T-SNE**.\n\n## Notebook Link:\n* [Happywhale: Data Distribution 🐋🐬](https://www.kaggle.com/awsaf49/happywhale-data-distribution)\n\n## Data Distribution:\nTo Plot **image embeddings** I've used using both **WandB** and **cuML**. Both of them are great libraries. Here are some of the visualizations,\n\n### Train Vs Test:\n
\"results-53-0\"
\n\n### Whales Vs Dolphin:\n
\"results-57-0\"
\n\n### Whales Species:\n
\"results-61-0\"
\n\n### Dolphin Species:\n
\"results-63-0\"
\n\n### Random **Train/Test**:\n\"trainvtest\"\n\n### Random **Whale/Dolphin**:\n\"wvsd\"\n\n### UMAP/PCA/TSNE using **WandB**\nWe can also plot embeddings using **wandb**. Here's some cool plots using **wandb**,\n\"w-b01\"\n\"w-b02\"\n\"w-b04\"", "votes": null }, { "id": "1676395", "postDate": "02/05/2022 00:14:56", "content": "

Great idea to use T-SNE. How do you interpret these plots? I'm having a hard time picking up on patterns.

", "rawMarkdown": "Great idea to use T-SNE. How do you interpret these plots? I'm having a hard time picking up on patterns.", "votes": null }, { "id": "1677432", "postDate": "02/05/2022 18:02:43", "content": "

As you may already know,

\n
T-SNE captures structure in the sense that neighboring points in the input space will tend to be neighbors in the low dimensional space. Hence, similar images will stay close and dissimilar images will stay far.\n
\n

Consider the Whale/Dolphin plot. Here background is Whale(red) and foreground is Dolphin(green). If we can see only red that means it's only Whale. If we can see green it could be either Dolphin(foreground) or Whale + Dolphin (background + foreground). If you look closely at this plot, you would find that there are some regions where we can only see whales (background -> red) which means no dolphins (color: green) are present having similar features. There are some small regions with only the color green where you might think it could be Whales + Dolphin or Dolphin but if you revert the background with foreground you can confirm that it is only Dolphin.

\n

But if you look at the Train/Test plot you won't find such a region for test data which means for almost every test sample we have a similar train sample.

", "rawMarkdown": "As you may already know, \n```\nT-SNE captures structure in the sense that neighboring points in the input space will tend to be neighbors in the low dimensional space. Hence, similar images will stay close and dissimilar images will stay far.\n```\nConsider the **Whale/Dolphin** plot. Here background is **Whale**(red) and foreground is **Dolphin**(green). If we can see only red that means it's only **Whale**. If we can see **green** it could be either **Dolphin**(foreground) or **Whale + Dolphin** (background + foreground). If you look closely at this plot, you would find that there are some regions where we can only see **whales** (background -> **red**) which means no **dolphins** (color: **green**) are present having similar features. There are some small regions with only the color **green** where you might think it could be **Whales + Dolphin** or **Dolphin** but if you revert the background with foreground you can confirm that it is only **Dolphin**.\n\nBut if you look at the **Train/Test** plot you won't find such a region for **test data** which means for almost every **test** sample we have a similar **train** sample.", "votes": null }, { "id": "1677738", "postDate": "02/06/2022 00:14:36", "content": "

One quick observation from a biologist: the whale/dolphin split here doesn't map to any meaningful biological (evolutionary or ecological) categories. Baleen whales (e.g., humpback, gray, or blue) and toothed whales (e.g., pilot whale, spinner dolphin, or spotted dolphin) would be a more meaningful biological distinction. That said, I'm not sure that this distinction would affect the look of your charts.

", "rawMarkdown": "One quick observation from a biologist: the whale/dolphin split here doesn't map to any meaningful biological (evolutionary or ecological) categories. Baleen whales (e.g., humpback, gray, or blue) and toothed whales (e.g., pilot whale, spinner dolphin, or spotted dolphin) would be a more meaningful biological distinction. That said, I'm not sure that this distinction would affect the look of your charts.", "votes": null }, { "id": "1677810", "postDate": "02/06/2022 02:52:16", "content": "

I wonder if that's the reason why whale and dolphin data distribution look similar…

", "rawMarkdown": "I wonder if that's the reason why **whale** and **dolphin** data distribution look similar...", "votes": null }, { "id": "1678909", "postDate": "02/06/2022 21:01:23", "content": "

Great visualization! Upvoted @awsaf49

", "rawMarkdown": "Great visualization! Upvoted @awsaf49", "votes": null }, { "id": "1678949", "postDate": "02/06/2022 21:37:12", "content": "

I wouldn't read too much into it. Other than the small regions, both whales and dolphins are distributed throughout the 2-d space. Below is a T-SNE plot of the MNIST handwritten digits showing clear separation of groups. How to Use t-SNE Effectively

\n

Tensorflow Embedding Projector
\n\"mnist\"

", "rawMarkdown": "I wouldn't read too much into it. Other than the small regions, both whales and dolphins are distributed throughout the 2-d space. Below is a T-SNE plot of the MNIST handwritten digits showing clear separation of groups. [How to Use t-SNE Effectively](https://distill.pub/2016/misread-tsne/) \n\n[Tensorflow Embedding Projector](https://projector.tensorflow.org/)\n![mnist](https://i.imgur.com/KbBswle.png)", "votes": null }, { "id": "1679171", "postDate": "02/07/2022 03:41:45", "content": "
Other than the small regions, both whales and dolphins are distributed throughout the 2-d space\n
\n

Yes, you are right, but in Train/Test there are no such small regions hence they have similar features …

", "rawMarkdown": "```\nOther than the small regions, both whales and dolphins are distributed throughout the 2-d space\n```\nYes, you are right, but in **Train/Test** there are no such **small regions** hence they have similar features ...", "votes": null }, { "id": "1682883", "postDate": "02/09/2022 12:42:54", "content": "

This is great im also hoping we can use NLP based algorithms to try and translate Dolphin Languages as they are creatures with high intellect amongst many in the Ocean.

", "rawMarkdown": "This is great im also hoping we can use NLP based algorithms to try and translate Dolphin Languages as they are creatures with high intellect amongst many in the Ocean.", "votes": null } ], "comments": [ { "id": 1676395, "author_name": "jpmiller", "author_url": "", "post_date": "02/05/2022 00:14:56", "content": "

Great idea to use T-SNE. How do you interpret these plots? I'm having a hard time picking up on patterns.

", "votes": null, "replies": [ { "id": 1677432, "author_name": "awsaf49", "author_url": "", "post_date": "02/05/2022 18:02:43", "content": "

As you may already know,

\n
T-SNE captures structure in the sense that neighboring points in the input space will tend to be neighbors in the low dimensional space. Hence, similar images will stay close and dissimilar images will stay far.\n
\n

Consider the Whale/Dolphin plot. Here background is Whale(red) and foreground is Dolphin(green). If we can see only red that means it's only Whale. If we can see green it could be either Dolphin(foreground) or Whale + Dolphin (background + foreground). If you look closely at this plot, you would find that there are some regions where we can only see whales (background -> red) which means no dolphins (color: green) are present having similar features. There are some small regions with only the color green where you might think it could be Whales + Dolphin or Dolphin but if you revert the background with foreground you can confirm that it is only Dolphin.

\n

But if you look at the Train/Test plot you won't find such a region for test data which means for almost every test sample we have a similar train sample.

", "votes": null, "replies": [] }, { "id": 1678949, "author_name": "jpmiller", "author_url": "", "post_date": "02/06/2022 21:37:12", "content": "

I wouldn't read too much into it. Other than the small regions, both whales and dolphins are distributed throughout the 2-d space. Below is a T-SNE plot of the MNIST handwritten digits showing clear separation of groups. How to Use t-SNE Effectively

\n

Tensorflow Embedding Projector
\n\"mnist\"

", "votes": null, "replies": [] }, { "id": 1679171, "author_name": "awsaf49", "author_url": "", "post_date": "02/07/2022 03:41:45", "content": "
Other than the small regions, both whales and dolphins are distributed throughout the 2-d space\n
\n

Yes, you are right, but in Train/Test there are no such small regions hence they have similar features …

", "votes": null, "replies": [] } ] }, { "id": 1677738, "author_name": "philtpatton", "author_url": "", "post_date": "02/06/2022 00:14:36", "content": "

One quick observation from a biologist: the whale/dolphin split here doesn't map to any meaningful biological (evolutionary or ecological) categories. Baleen whales (e.g., humpback, gray, or blue) and toothed whales (e.g., pilot whale, spinner dolphin, or spotted dolphin) would be a more meaningful biological distinction. That said, I'm not sure that this distinction would affect the look of your charts.

", "votes": null, "replies": [ { "id": 1677810, "author_name": "awsaf49", "author_url": "", "post_date": "02/06/2022 02:52:16", "content": "

I wonder if that's the reason why whale and dolphin data distribution look similar…

", "votes": null, "replies": [] } ] }, { "id": 1678909, "author_name": "nebipeker", "author_url": "", "post_date": "02/06/2022 21:01:23", "content": "

Great visualization! Upvoted @awsaf49

", "votes": null, "replies": [] }, { "id": 1682883, "author_name": "muhammadammarjamshed", "author_url": "", "post_date": "02/09/2022 12:42:54", "content": "

This is great im also hoping we can use NLP based algorithms to try and translate Dolphin Languages as they are creatures with high intellect amongst many in the Ocean.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1673977": "One of the key challenges of this competition is that we've to identify `new_indiviual` even though they don't appear on the **train** data. My initial thought was that `new_individual` may look very different as they don't appear on the **train** data. But **image embeddings** tell a different story. I've published a notebook to demonstrate this analysis using **T-SNE**.\n\n## Notebook Link:\n* [Happywhale: Data Distribution 🐋🐬](https://www.kaggle.com/awsaf49/happywhale-data-distribution)\n\n## Data Distribution:\nTo Plot **image embeddings** I've used using both **WandB** and **cuML**. Both of them are great libraries. Here are some of the visualizations,\n\n### Train Vs Test:\n
\"results-53-0\"
\n\n### Whales Vs Dolphin:\n
\"results-57-0\"
\n\n### Whales Species:\n
\"results-61-0\"
\n\n### Dolphin Species:\n
\"results-63-0\"
\n\n### Random **Train/Test**:\n\"trainvtest\"\n\n### Random **Whale/Dolphin**:\n\"wvsd\"\n\n### UMAP/PCA/TSNE using **WandB**\nWe can also plot embeddings using **wandb**. Here's some cool plots using **wandb**,\n\"w-b01\"\n\"w-b02\"\n\"w-b04\"", "1676395": "Great idea to use T-SNE. How do you interpret these plots? I'm having a hard time picking up on patterns.", "1677432": "As you may already know, \n```\nT-SNE captures structure in the sense that neighboring points in the input space will tend to be neighbors in the low dimensional space. Hence, similar images will stay close and dissimilar images will stay far.\n```\nConsider the **Whale/Dolphin** plot. Here background is **Whale**(red) and foreground is **Dolphin**(green). If we can see only red that means it's only **Whale**. If we can see **green** it could be either **Dolphin**(foreground) or **Whale + Dolphin** (background + foreground). If you look closely at this plot, you would find that there are some regions where we can only see **whales** (background -> **red**) which means no **dolphins** (color: **green**) are present having similar features. There are some small regions with only the color **green** where you might think it could be **Whales + Dolphin** or **Dolphin** but if you revert the background with foreground you can confirm that it is only **Dolphin**.\n\nBut if you look at the **Train/Test** plot you won't find such a region for **test data** which means for almost every **test** sample we have a similar **train** sample.", "1677738": "One quick observation from a biologist: the whale/dolphin split here doesn't map to any meaningful biological (evolutionary or ecological) categories. Baleen whales (e.g., humpback, gray, or blue) and toothed whales (e.g., pilot whale, spinner dolphin, or spotted dolphin) would be a more meaningful biological distinction. That said, I'm not sure that this distinction would affect the look of your charts.", "1677810": "I wonder if that's the reason why **whale** and **dolphin** data distribution look similar...", "1678909": "Great visualization! Upvoted @awsaf49", "1678949": "I wouldn't read too much into it. Other than the small regions, both whales and dolphins are distributed throughout the 2-d space. Below is a T-SNE plot of the MNIST handwritten digits showing clear separation of groups. [How to Use t-SNE Effectively](https://distill.pub/2016/misread-tsne/) \n\n[Tensorflow Embedding Projector](https://projector.tensorflow.org/)\n![mnist](https://i.imgur.com/KbBswle.png)", "1679171": "```\nOther than the small regions, both whales and dolphins are distributed throughout the 2-d space\n```\nYes, you are right, but in **Train/Test** there are no such **small regions** hence they have similar features ...", "1682883": "This is great im also hoping we can use NLP based algorithms to try and translate Dolphin Languages as they are creatures with high intellect amongst many in the Ocean." }, "source": "meta" }