Notebook works backwards through trained CNN network to create best approximation of input - given a selected bird classification. You can now hear what the solution “thinks” each bird sounds like after it has been trained. Notebook uses similar technique of listening to recreated input to explore feature extraction, internal layers of CNN, and the whole solution - to see what information is thrown away and which is retained during inference. I hope you find the notebook fun and informative!
\nhttps://www.kaggle.com/code/pnussbaum/v15h-birdclef2023-mindreader
\n-Paul Alton Nussbaum
", "messages": [ { "id": "2234093", "postDate": "04/24/2023 21:26:44", "content": "Notebook works backwards through trained CNN network to create best approximation of input - given a selected bird classification. You can now hear what the solution “thinks” each bird sounds like after it has been trained. Notebook uses similar technique of listening to recreated input to explore feature extraction, internal layers of CNN, and the whole solution - to see what information is thrown away and which is retained during inference. I hope you find the notebook fun and informative!
\nhttps://www.kaggle.com/code/pnussbaum/v15h-birdclef2023-mindreader
\n-Paul Alton Nussbaum
", "rawMarkdown": "Notebook works backwards through trained CNN network to create best approximation of input - given a selected bird classification. You can now hear what the solution “thinks” each bird sounds like after it has been trained. Notebook uses similar technique of listening to recreated input to explore feature extraction, internal layers of CNN, and the whole solution - to see what information is thrown away and which is retained during inference. I hope you find the notebook fun and informative!\n\nhttps://www.kaggle.com/code/pnussbaum/v15h-birdclef2023-mindreader\n\n-Paul Alton Nussbaum", "votes": null }, { "id": "2245464", "postDate": "05/04/2023 11:43:35", "content": "Thanks to everyone who took a look at and upvoted the \"mindreader\" notebook! It is greatly appreciated.
\nI coined the term \"Reading the Robot Mind\" to explain the process of presenting the internal workings of an Artificial Neural Network, or Artificial Intelligence (AI) system, in a way that is intuitive to Subject Matter Experts (SME).
\nI just wanted to wrap it up with the follow-on notebooks, all made public for your convenience:
\nhttps://www.kaggle.com/code/pnussbaum/v15h-birdclef2023-mindreader - The notebook I first mentioned, and the best notebook to experiment with. This notebook focuses on the Segmentation and Feature Extraction aspects of the AI solution, allowing users to make modifications (Mel Scale Spectrum, number of resultant frequency bands, MFCC, number of resultant coefficients) and see and hear how much information is retained. It also allows the user to train a simple Convolutional Neural Network (CNN) AI solution and see and hear how much information is retained at each layer - using only a limited number of birds to speed up experimentation. Reading the robot mind learnings include selection of a feature extraction algorithm whereby SME can see and hear that important information has not been discarded, and can also see similarities in feature visualizations (spectrograms) for the same type of birds, while also spotting visual differences between different bird types. The final method used 5 second segments, each converted into an overlapping sequence of 32 Mel Scaled frequency bands.
https://www.kaggle.com/code/pnussbaum/v15h-all-birdclef2023-mindreader - This notebook allows the user to use their final decision related to segmentation and feature extraction, and convert and save the BirdClef2023 data into this format. It also allows a short amount of training of a CNN on this data; once again, allowing the SME to see and hear how much data is retained at each layer - this time using the entire data set. Learnings include identification of specific layers where important information seems to be discarded. For example, it was noted that the recreation of the input seemed to degrade significantly at certain layers of the network, and so these layers were specifically modified to improve the retention of important information (in the form of additional nerons for dense layers and additional filters for convolutional layers).
https://www.kaggle.com/code/pnussbaum/v16e-gpu-all-birdclef2023-mindreader - This notebook uses the final decisions noted above, and trains the entire CNN for a longer period of time, achieving better accuracy, and saving the trained AI system. There were several iterations of this notebook, with improvements made based on the reading the robot mind system, but also traditional troubleshooting techniques. An example of one of the traditional AI troubleshooting techniques was the identification of overfitting. This was remediated through the use of data augmentation (shifting the data in time by a small random value during training, etc.)
https://www.kaggle.com/code/pnussbaum/v17b-all-birdclef2023-mindreader - This notebook brings all of this together for the sake of the contest submission, as well as inference analysis and trouleshooting. The notebook lets you hear, visualize, and compare data every step and layer on the way. It allows comparison of samples from the same type of bird (so you can see and hear if they are similar) and different types of birds (so you can see and hear if they are different). Finally, this notebook provides the ability to work backwards through the system from a manually forced output, and let the user see and hear a best estimation of what the trained AI \"thinks\" that bird type sounds like. Learnings include the ability to hear elements of unique aspects particular to that bird type (short sequences of sound) all mashed together, with a good deal of extraneous noise mixed in.
Thanks to everyone who took a look at and upvoted the \"mindreader\" notebook! It is greatly appreciated.
\nI coined the term \"Reading the Robot Mind\" to explain the process of presenting the internal workings of an Artificial Neural Network, or Artificial Intelligence (AI) system, in a way that is intuitive to Subject Matter Experts (SME).
\nI just wanted to wrap it up with the follow-on notebooks, all made public for your convenience:
\nhttps://www.kaggle.com/code/pnussbaum/v15h-birdclef2023-mindreader - The notebook I first mentioned, and the best notebook to experiment with. This notebook focuses on the Segmentation and Feature Extraction aspects of the AI solution, allowing users to make modifications (Mel Scale Spectrum, number of resultant frequency bands, MFCC, number of resultant coefficients) and see and hear how much information is retained. It also allows the user to train a simple Convolutional Neural Network (CNN) AI solution and see and hear how much information is retained at each layer - using only a limited number of birds to speed up experimentation. Reading the robot mind learnings include selection of a feature extraction algorithm whereby SME can see and hear that important information has not been discarded, and can also see similarities in feature visualizations (spectrograms) for the same type of birds, while also spotting visual differences between different bird types. The final method used 5 second segments, each converted into an overlapping sequence of 32 Mel Scaled frequency bands.
https://www.kaggle.com/code/pnussbaum/v15h-all-birdclef2023-mindreader - This notebook allows the user to use their final decision related to segmentation and feature extraction, and convert and save the BirdClef2023 data into this format. It also allows a short amount of training of a CNN on this data; once again, allowing the SME to see and hear how much data is retained at each layer - this time using the entire data set. Learnings include identification of specific layers where important information seems to be discarded. For example, it was noted that the recreation of the input seemed to degrade significantly at certain layers of the network, and so these layers were specifically modified to improve the retention of important information (in the form of additional nerons for dense layers and additional filters for convolutional layers).
https://www.kaggle.com/code/pnussbaum/v16e-gpu-all-birdclef2023-mindreader - This notebook uses the final decisions noted above, and trains the entire CNN for a longer period of time, achieving better accuracy, and saving the trained AI system. There were several iterations of this notebook, with improvements made based on the reading the robot mind system, but also traditional troubleshooting techniques. An example of one of the traditional AI troubleshooting techniques was the identification of overfitting. This was remediated through the use of data augmentation (shifting the data in time by a small random value during training, etc.)
https://www.kaggle.com/code/pnussbaum/v17b-all-birdclef2023-mindreader - This notebook brings all of this together for the sake of the contest submission, as well as inference analysis and trouleshooting. The notebook lets you hear, visualize, and compare data every step and layer on the way. It allows comparison of samples from the same type of bird (so you can see and hear if they are similar) and different types of birds (so you can see and hear if they are different). Finally, this notebook provides the ability to work backwards through the system from a manually forced output, and let the user see and hear a best estimation of what the trained AI \"thinks\" that bird type sounds like. Learnings include the ability to hear elements of unique aspects particular to that bird type (short sequences of sound) all mashed together, with a good deal of extraneous noise mixed in.