{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Learning on Herbarium Kaggle Loop
\n\n\nSince the Scope Topics of the Annual Workshop on Fine-Grained Visual Categorization at CVPR 2022 include:\n\nFine-grained categorization with humans in the loop\n\nEmbedding human experts’ knowledge into computational models\n\nMachine teaching\n\nInterpretable fine-grained models\n\nLet's read some insights about the topics above.","metadata":{}},{"cell_type":"markdown","source":"youtube.com","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd\nfrom pathlib import Path\nimport os.path\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nimport seaborn as sns\nfrom time import perf_counter\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import classification_report,accuracy_score\nfrom IPython.display import Markdown, display\n\ndef printmd(string):\n # Print with Markdowns \n display(Markdown(string))","metadata":{"execution":{"iopub.status.busy":"2022-02-18T17:58:42.706763Z","iopub.execute_input":"2022-02-18T17:58:42.70714Z","iopub.status.idle":"2022-02-18T17:58:49.150022Z","shell.execute_reply.started":"2022-02-18T17:58:42.70705Z","shell.execute_reply":"2022-02-18T17:58:49.148936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Human-in-the-Loop and ML
\n\nHuman-in-the-Loop and Machine Learning (Appen Blog)\n\n\"Human-in-the-loop (HITL) is a branch of AI that leverages both human and machine intelligence to create machine learning models. In a traditional human-in-the-loop approach, people are involved in a virtuous circle where they train, tune, and test a particular algorithm. Generally, it works like this:\"\n\n\"Label data. This gives a model high quality (and high quantities of) training data. A machine learning algorithm learns to make decisions from this data.\"\n\n\"Tune the model. This can happen in several different ways, but commonly, humans will score data to account for overfitting, to teach a classifier about edge cases, or new categories in the model’s purview.\"\n\n\"Test and validate a model by scoring its outputs, especially in places where an algorithm is unconfident about a judgment or overly confident about an incorrect decision.\"\n\n\"Those actions comprises a continuous feedback loop. Human-in-the-loop machine learning means taking each of these training, tuning, and testing tasks and feeding them back into the algorithm so it gets smarter, more confident, and more accurate. This can be especially effective when the model selects what it needs to learn next–known as active learning–and you send that data to human annotators for training.\"\n\nhttps://appen.com/blog/human-in-the-loop/","metadata":{}},{"cell_type":"markdown","source":"manningbooks.medium.com","metadata":{}},{"cell_type":"code","source":"#Code by Datalira https://www.kaggle.com/databeru/classify-bricks-compare-transfer-learning-model/notebook\n\n# Create a list with the filepaths for training and testing\n#train_dir = Path('../input/herbarium-2022-fgvc9/train_images')\n#test_dir = '../input/herbarium-2022-fgvc9/test_images'\ndir_ = Path('../input/herbarium-2022-fgvc9/train_images')\n\nfile_paths = list(dir_.glob(r'**/*.jpg'))\nfile_paths = [str(x) for x in file_paths]\ndf = pd.DataFrame({'Filepath':file_paths})","metadata":{"execution":{"iopub.status.busy":"2022-02-18T17:59:21.526676Z","iopub.execute_input":"2022-02-18T17:59:21.526904Z","iopub.status.idle":"2022-02-18T18:03:04.910438Z","shell.execute_reply.started":"2022-02-18T17:59:21.526878Z","shell.execute_reply":"2022-02-18T18:03:04.909576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Human-in-the-Loop and Active Learning
\n\nDifference between human-in-the-loop and active learning\n\n\"Active learning generally refers to the humans handling low confidence units and feeding those back into the model. Human-in-the-loop is broader, encompassing active learning approaches as well as the creation of data sets through human labeling. Additionally, HitL can sometimes (though rarely) refer to people simply validating (or invalidating) an output without feeding those judgments back to the model.\"\n\nhttps://appen.com/blog/human-in-the-loop/","metadata":{}},{"cell_type":"code","source":"#Code by Datalira https://www.kaggle.com/databeru/classify-bricks-compare-transfer-learning-model/notebook\n\ndef get_label(string):\n string = ' '.join(string.split('/')[-1].replace('.jpg', '').split(' ')[1:-1])\n string = string.lower()\n return string\n\n# Retrieve the label from the path of the pictures\ndf['Label'] = df['Filepath'].apply(lambda x: get_label(x))","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:03:39.436015Z","iopub.execute_input":"2022-02-18T18:03:39.43658Z","iopub.status.idle":"2022-02-18T18:03:40.323881Z","shell.execute_reply.started":"2022-02-18T18:03:39.436531Z","shell.execute_reply":"2022-02-18T18:03:40.322923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"More training data. Better performance
\n\nMore training data = better performance\n\n\"HITL refers to systems that allow humans to give direct feedback to a model for predictions below a certain level of confidence.\"\n\n\"In practice, you need to determine what level of confidence is acceptable for the process: If it is ok to have wrong predictions \"slipping through\", you can set threshold rather low – which, in turn, requires much less manual intervention through human labor. In other cases, you want to be sure that the system only records \"correct\" predictions.\"\n\n\"Why can't we just use better algorithms to achieve higher confidence?\"\n\n\"The field of AI has seen great technological advances . The fundamental problem of this is that training data is hard to get by as it requires human expertise. And while there are many public datasets available, they generally don't exist for your specific problems. Hence, they have to be created.\"\n\n\"In order not to spend 3 years building a dataset, it is possible to already start training a model and using it sooner. In many cases, this already leads to considerable productivity gains.\"\n\nhttps://levity.ai/blog/human-in-the-loop\n\nImproving the accuracy of rare datasets
\n\nImproving the accuracy of rare datasets\n\n\"Where there is a lack of data, machine learning models are not very useful.\"\n\n\"Healthcare as an example. A 2018 Stanford study found that a human-in-the-loop AI model works better than AI on its own or human doctors on their own.\"\n\n\"These systems can help improve accuracy while maintaining human-level standards of work. This can apply to many industries and change the way people work all across the world!\"\n\nhttps://levity.ai/blog/human-in-the-loop","metadata":{}},{"cell_type":"code","source":"#Code by Datalira https://www.kaggle.com/databeru/classify-bricks-compare-transfer-learning-model/notebook\n\n# Display some pictures of the dataset\nfig, axes = plt.subplots(nrows=4, ncols=6, figsize=(15, 7),\n subplot_kw={'xticks': [], 'yticks': []})\n\nfor i, ax in enumerate(axes.flat):\n ax.imshow(plt.imread(df.Filepath[i]))\n ax.set_title(df.Label[i], fontsize = 15)\nplt.tight_layout(pad=0.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:03:45.792593Z","iopub.execute_input":"2022-02-18T18:03:45.79284Z","iopub.status.idle":"2022-02-18T18:03:48.794341Z","shell.execute_reply.started":"2022-02-18T18:03:45.792816Z","shell.execute_reply":"2022-02-18T18:03:48.793523Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Human-in-the-loop and Deep Learning
\n\nHumans and machines - Human-in-the-loop Deep Learning approach.\n\n\"In supervised learning, experts use labeled data sets to train algorithms to produce appropriate functions. These can then help to map new examples. Doing this will allow the algorithm to correctly determine functions for unlabeled data.\"\n\n\"In unsupervised learning, unlabeled datasets are fed to the algorithms. Thus, they need to learn on their own to find a structure in the unlabeled data and memorize it accordingly. This falls under the human-in-the-loop deep learning approach.\"\n\nhttps://levity.ai/blog/human-in-the-loop","metadata":{}},{"cell_type":"code","source":"#Code by Datalira https://www.kaggle.com/databeru/classify-bricks-compare-transfer-learning-model/notebook\n\n# Display the number of pictures of each category\nvc = df['Label'].value_counts()\nplt.figure(figsize=(20,5))\nsns.barplot(x = sorted(vc.index), y = vc, palette = \"nipy_spectral\")\nplt.title(\"Number of pictures of each category\", fontsize = 15)\nplt.xticks(rotation=90)\nplt.show()","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2022-02-18T18:05:22.911365Z","iopub.execute_input":"2022-02-18T18:05:22.911654Z","iopub.status.idle":"2022-02-18T18:05:23.095168Z","shell.execute_reply.started":"2022-02-18T18:05:22.911626Z","shell.execute_reply":"2022-02-18T18:05:23.094121Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Not very useful chart since we have only one category?","metadata":{}},{"cell_type":"markdown","source":"A Survey of Human-in-the-loop for Machine Learning
\n\nA Survey of Human-in-the-loop for Machine Learning\n\nAuthors: Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, Liang He\n\narXiv:2108.00941 \n\n\"Human-in-the-loop aims to train an accurate prediction model with minimum cost by integrating human knowledge and experience. Humans can provide training data for machine learning applications and directly accomplish tasks that are hard for computers in the pipeline with the help of machine-based approaches.\"\n\n\" In this paper, the authors surveyed existing works on human-in-the-loop from a data perspective and classify them into three categories with a progressive relationship: (1) the work of improving model performance from data processing, (2) the work of improving model performance through interventional model training, and (3) the design of the system independent human-in-the-loop. \"\n\n\"That survey intends to provide a high-level summarization for human-in-the-loop and motivates interested readers to consider approaches for designing effective human-in-the-loop solutions.\"\n\nhttps://arxiv.org/abs/2108.00941","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:05:36.709709Z","iopub.execute_input":"2022-02-18T18:05:36.710064Z","iopub.status.idle":"2022-02-18T18:05:36.713221Z","shell.execute_reply.started":"2022-02-18T18:05:36.710039Z","shell.execute_reply":"2022-02-18T18:05:36.712712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Accelerating the AI Lifecycle
\n\nAccelerating the AI Lifecycle\n\n\"The AI lifecycle moves from proof of concept to proof of scale, then to model in production. This is a cyclical process, and people play an important role. Humans in the loop (HITL) inspect, validate, and make changes to algorithms to improve outcomes. They also collect, label, and conduct quality control (QC) on data. The benefits of humans in the loop begin with model development and extend across the AI lifecycle, from proof of concept to model in production.\"\n\n\"Poor utilization of people in the AI lifecycle is costly and yields low-quality data and poor model performance. It is best to consider workforce early in the AI lifecycle, and before model development begins, to achieve higher quality outcomes.\"\n\nhttps://www.cloudfactory.com/human-in-the-loop","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import backend as K\nfrom tensorflow.keras.layers import Dense, Activation,Dropout,Conv2D, MaxPooling2D,BatchNormalization, Flatten\nfrom tensorflow.keras.optimizers import Adam, Adamax\nfrom tensorflow.keras.metrics import categorical_crossentropy\nfrom tensorflow.keras import regularizers\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Model, load_model, Sequential","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:05:41.406796Z","iopub.execute_input":"2022-02-18T18:05:41.407021Z","iopub.status.idle":"2022-02-18T18:05:42.48819Z","shell.execute_reply.started":"2022-02-18T18:05:41.406997Z","shell.execute_reply":"2022-02-18T18:05:42.486984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Embedding Human Knowledge into DNN via Attention Map
\n\nEmbedding Human Knowledge into Deep Neural Network via Attention Map\n\nAuthors: Masahiro Mitsuhara, Hiroshi Fukui, Yusuke Sakashita, Takanori Ogata, Tsubasa Hirakawa, Takayoshi Yamashita, Hironobu Fujiyoshi\n\nCitation: arXiv:1905.03540 \n\n\"In this work, the authors aimed to realize a method for embedding human knowledge into deep neural networks. While the conventional method to embed human knowledge has been applied for non-deep machine learning, it is challenging to apply it for deep learning models due to the enormous number of model parameters.\"\n\n\"In this paper, they proposed a fine-tuning method that utilizes a single-channel attention map which is manually edited by a human expert. Their fine-tuning method can train a network so that the output attention map corresponds to the edited ones. As a result, the fine-tuned network can output an attention map that takes into account human knowledge.\"\n\n\"Experimental results with ImageNet, CUB-200-2010, and IDRiD demonstrated that it is possible to obtain a clear attention map for a visual explanation and improve the classification performance. Their findings can be a novel framework for optimizing networks through human intuitive editing via a visual interface and suggest new possibilities for human-machine cooperation in addition to the improvement of visual explanations.\"\n\nhttps://arxiv.org/abs/1905.03540","metadata":{}},{"cell_type":"code","source":"import shutil\nimport time\nimport cv2 as cv2\nfrom tqdm import tqdm\nfrom sklearn.model_selection import train_test_split\nimport matplotlib.pyplot as plt\nfrom matplotlib.pyplot import imshow\n\nimport seaborn as sns\nsns.set_style('darkgrid')\nfrom PIL import Image\nfrom sklearn.metrics import confusion_matrix, classification_report\nfrom IPython.core.display import display, HTML","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:05:48.304634Z","iopub.execute_input":"2022-02-18T18:05:48.304881Z","iopub.status.idle":"2022-02-18T18:05:48.743016Z","shell.execute_reply.started":"2022-02-18T18:05:48.304855Z","shell.execute_reply":"2022-02-18T18:05:48.741875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Integrating Machine Learning with Human Knowledge
\n\nIntegrating Machine Learning with Human Knowledge \n\nAuthors: Changyu Deng ; Xunbi Ji; Colton Rainey; Jianyu Zhang ; Wei Lu\n\nhttps://doi.org/10.1016/j.isci.2020.101656\n\nHIGHLIGHTS\n\nIntegrating knowledge into machine learning delivers superior performance\n\nKnowledge is categorized and its representations are presented\n\nVarious methods to bridge human knowledge and machine learning are shown\n\nSuggestions on approaches and perspectives on future research directions are provided\n\n\"Multitask Learning\"\n\n\"Humans do not learn individual tasks in a linear sequence, but they learn several tasks simultaneously. This efficient behavior is replicated in machine learning with multitask learning (MTL). MTL shares knowledge between tasks so they are all learned simultaneously with higher overall performance .\"\n\nRead all these in that paper ( Model Assumptions, Preliminaries of Probabilistic ML, Deterministic Assumptions, Network Architecture, Sy,mmetry of Convolutional Neural Networks, Design of Neuron Connections, Data Augmentation, Simulation, Reinforcement Learning,Active Learning, Interactive Visual Analytics, Parameter Initialization, Transfer Learning.\n\n\"As humans create faster and more accurate knowledge-based models to simulate the world, using simulations to acquire a large amount of data becomes an increasingly efficient method for machine learning. The primary advantage of simulations is the ability to gather a large amount of data when experimentally would be costly, time consuming, or even dangerous\".\n\n\"If using neural networks, tailor the architecture to be suitable for the tasks. If possible, incorporate some known properties, such as Symmetry of Convolutional Neural Networks. Logic, equations, and temporal nature can be, respectively, reflected in the structure of networks by, for instance, combining with symbolic AI, designing special layers/architectures, and using RNNs.\"\n\n\"Design algorithms to include humans in the loop. The interaction between machine and environment can be modeled and optimized in Reinforcement Learning . Humans can be asked to label data or provide distribution (Active Learning). Interactive Visual Analytics can be used to help humans understand machine learning results and then adjust models during or after training.\"\n\n\"The authors can regulate the intermediate results or network layers to produce models more understandable and controllable by humans.\"\n\n\"Designing and implementing machine learning algorithms is an iterative process. This requires humans to analyze the models and knowledge integration to take advantage of human understanding of the real world. That review may help current and prospective users of machine learning to understand these fields and inspire them to build more efficient models.\"\n\nhttps://www.sciencedirect.com/science/article/pii/S2589004220308488","metadata":{}},{"cell_type":"markdown","source":"#Plan B","metadata":{}},{"cell_type":"code","source":"#Code by Gerry https://www.kaggle.com/gpiosenka/gym-f1-score-100/data?select=MontezumaRevenge\n\n#Supress annoying tensorflow warnings\n\nimport logging\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' # FATAL\nlogging.getLogger('tensorflow').setLevel(logging.FATAL)","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:05:53.68815Z","iopub.execute_input":"2022-02-18T18:05:53.688389Z","iopub.status.idle":"2022-02-18T18:05:53.694004Z","shell.execute_reply.started":"2022-02-18T18:05:53.688363Z","shell.execute_reply":"2022-02-18T18:05:53.692855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Interpretable and Fine-Grained Visual Explanations for CNN
\n\nInterpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks\n\nAuthors: Jörg Wagner, Jan Mathias Köhler, Tobias Gindele, Leon Hetzel, Jakob Thaddäus Wiedemer, Sven Behnke - arXiv:1908.02686v1\n\n\"To verify and validate networks, it is essential to gain insight into their decisions, limitations as well as possible shortcomings of training data. In this work, the authors proposed a post-hoc, optimization based visual explanation method, which highlights the evidence in the input image for a specific prediction.\"\n\n\"Their approach is based on a novel technique to defend against adversarial evidence (i.e. faulty evidence due to artefacts) by filtering gradients during optimization. The defense does not depend on human-tuned parameters. It enables explanations which are both fine-grained and preserve the characteristics of images, such as edges and colors.\"\n\n\"The explanations are interpretable, suited for visualizing detailed evidence and can be tested as they are valid model inputs. The authors qualitatively and quantitatively evaluated their approach on a multitude of models and datasets.\"\n\nhttps://arxiv.org/abs/1908.02686","metadata":{}},{"cell_type":"code","source":"#Code by Gerry https://www.kaggle.com/gpiosenka/gym-f1-score-100/data?select=MontezumaRevenge\n\n#input an image\n\nfpath=r'../input/herbarium-2022-fgvc9/train_images/004/04/00404__001.jpg'\nimg=plt.imread(fpath)\nprint (img.shape)\nimshow(img);","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-18T18:05:58.282852Z","iopub.execute_input":"2022-02-18T18:05:58.283774Z","iopub.status.idle":"2022-02-18T18:05:58.587524Z","shell.execute_reply.started":"2022-02-18T18:05:58.283708Z","shell.execute_reply":"2022-02-18T18:05:58.586587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Interpretable and Accurate Fine-grained Recognition via Region Grouping
\n\n\nInterpretable and Accurate Fine-grained Recognition via Region Grouping\nAuthors: Zixuan Huang, Yin Li\n\n\"The authors presented an interpretable deep model for fine-grained visual recognition. At the core of our method lies the integration of region-based part discovery and attribution within a deep neural network.\"\n\n\"Their model is trained using image-level object labels, and provides an interpretation of its results via the segmentation of object parts and the identification of their contributions towards classification.\"\n\n\"To facilitate the learning of object parts without direct supervision, they explored a simple prior of the occurrence of object parts. The authors demonstrated that this prior, when combined with their region-based part discovery and attribution, leads to an interpretable model that remains highly accurate. Their model is evaluated on major fine-grained recognition datasets, including CUB-200, CelebA and iNaturalist.\n\n\"Their results compared favourably to state-of-the-art methods on classification tasks, and outperformed previous approaches on the localization of object parts.\"\n\nPART DISCOVERY for FINE-GRAINED RECOGNITION.\n\n\"Identifying discriminative object parts is important for fine-grained classification. For example, bounding box or landmark annotations can be used to learn object parts for fine-grained classification.\"\n\n\"To avoid costly annotation of object parts, several recent works focused on unsupervised or weakly-supervised part learning using deep models. Spectral clustering on convolutional filters was performed to find representative filters for parts. It was proposed to learn a bank of\nconvolutional filters that capture class-specific object parts.\"\n\n\"Moreover, attention models have also been explored extensively for learning parts. It was made use of reinforcement learning to select region proposals for fine-grained classification.\"\n\nTo read about the results: Results on CUB-200-2011 and Results on iNaturalist2017 and their Interpretability.\nhttps://openaccess.thecvf.com/content_CVPR_2020/papers/Huang_Interpretable_and_Accurate_Fine-grained_Recognition_via_Region_Grouping_CVPR_2020_paper.pdf ","metadata":{}},{"cell_type":"code","source":"#Code by Gerry https://www.kaggle.com/gpiosenka/gym-f1-score-100/data?select=MontezumaRevenge\n\nsdir=r'../input/herbarium-2022-fgvc9'\nfilepaths=[]\nlabels=[]\nclasslist=os.listdir(sdir) \nfor klass in classlist:\n classpath=os.path.join(sdir)#I removed klass after sdir\n classpath=os.path.join(classpath, 'train_images')\n flist=os.listdir(classpath)\n for f in flist:\n fpath=os.path.join(classpath,f) \n filepaths.append(fpath)\n labels.append(klass)\n Fseries= pd.Series(filepaths, name='filepaths')\n Lseries=pd.Series(labels, name='labels') \n df=pd.concat([Fseries, Lseries], axis=1) \n \nprint('df length: ', len(df) )\nbalance=df['labels'].value_counts()\nprint (balance)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-18T18:06:04.076144Z","iopub.execute_input":"2022-02-18T18:06:04.076394Z","iopub.status.idle":"2022-02-18T18:06:04.111277Z","shell.execute_reply.started":"2022-02-18T18:06:04.076369Z","shell.execute_reply":"2022-02-18T18:06:04.110574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Machine Teaching, Machine Learning and Optimal Education
\n\n\nMachine Teaching: An Inverse Problem to Machine Learning and an Approach Toward Optimal Education\nAuthor: Xiaojin Zhu\n\n\"What is machine teaching? Consider a “student” who is a machine learning algorithm, for example, a Support Vector Machine (SVM) or kmeans clustering. Now consider a “teacher” who wants the student to learn a target model θ. For example, θ can be a specific hyperplane in SVM, or the location of the k centroids in kmeans. The teacher knows θ and the student’s\nlearning algorithm, and teaches by giving the student training examples. Machine teaching aims to design the optimal training set D.\"\n\n\" What do you mean by optimal? One definition is the cardinality of D: the smaller |D| is,\nthe better. But there are other definitions as we shall see.\"\n\n\" If we already know the true model θ , why bother training a learner? The applications are such that the teacher and the learner are separate entities, and the teacher cannot directly “hard\nwire” the learner. One application is education where the learner is a human student. A more sinister “application” is security where the learner is an adaptive spam filter, and the “teacher” is a hacker who wishes to change the filtering behavior by sending the spam filter specially designed messages. Regardless of the intention, machine teaching aims to maximally influence the learner via optimal training data.\"\n\nApplying Machine Teaching:\n\n\"Optimization: Despite our early successes in certain teaching settings, solving for the optimal training data D is still difficult in general. The author expected that many tools developed in the optimization community can be brought to bear on difficult problems.\"\n\n\"Theory: Machine teaching originated from the theoretical study of teaching dimension. It is important to understand the theoretical properties of the optimal training set under more general teaching settings. The author speculated that information theory may be a suitable tool here: the teacher is the encoder, the learner is the decoder, and the message is the target model. But there is a twist: the decoder is not ideal. It is specified by whatever machine learning algorithm it runs.\"\n\n\"Education: Arguably more complex, education first needs to identify computable cognitive models of the student. Existing intelligent tutoring systems are a good place to start: with a little effort, one may hypothesize the inner works of the student black-box.\"\n\n\"Novel application: Consider computer security. As mentioned earlier, machine teaching also describes the optimal attack strategy if a hacker wants to influence a learning agent, see and the references therein. The question is, knowing the optimal attack strategy predicted by machine teaching, can we effectively defend the learning agent? There may be other serendipitous applications of machine teaching besides education and computer security. \"\n\nhttps://pages.cs.wisc.edu/~jerryzhu/machineteaching/pub/MachineTeachingAAAI15.pdf","metadata":{}},{"cell_type":"code","source":"#Code by Gerry https://www.kaggle.com/gpiosenka/gym-f1-score-100/data?select=MontezumaRevenge\n\n#Separate df into train_df, test_df, valid_df\n\ntrain_split=.96\nvalid_split=.02\ndummy_split=valid_split/(1-train_split)\ntrain_df, dummy_df=train_test_split(df, train_size=train_split, shuffle=True, random_state=123)\nvalid_df, test_df=train_test_split(dummy_df, train_size=dummy_split, shuffle=True, random_state=123)\nprint('train_df length: ', len(train_df), ' test_df length: ', len(test_df), ' valid_df length: ', len(valid_df))","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-18T18:06:09.909075Z","iopub.execute_input":"2022-02-18T18:06:09.909492Z","iopub.status.idle":"2022-02-18T18:06:09.922668Z","shell.execute_reply.started":"2022-02-18T18:06:09.909465Z","shell.execute_reply":"2022-02-18T18:06:09.921682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by Gerry https://www.kaggle.com/gpiosenka/gym-f1-score-100/data?select=MontezumaRevenge\n\n#Limit number of samples per class in train_df to 600 to reduce training time\n\nprint ('original number of classes: ', len(df['labels'].unique())) \nsize=5 #Original number of classes is 5 (on the code was 600)\nsamples=[]\ngroup=train_df.groupby('labels')\nfor label in train_df['labels'].unique():\n Lgroup=group.get_group(label)\n count=int(Lgroup['labels'].value_counts()) \n \n if count>=56:\n sample=Lgroup.sample(size, axis=0)\n samples.append(sample) \ntrain_df=pd.concat(samples, axis=0).reset_index(drop=True)\nprint (len(train_df))\nprint ('final number of classes: ', len(train_df['labels'].unique())) \nprint (train_df['labels'].value_counts()) ","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-18T18:06:14.767296Z","iopub.execute_input":"2022-02-18T18:06:14.767597Z","iopub.status.idle":"2022-02-18T18:06:14.793968Z","shell.execute_reply.started":"2022-02-18T18:06:14.767568Z","shell.execute_reply":"2022-02-18T18:06:14.792792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by Gerry https://www.kaggle.com/gpiosenka/gym-f1-score-100/data?select=MontezumaRevenge\n\n#create train, test, valid generators\n\nheight=200\nwidth=160\nchannels=3\nbatch_size=40\nimg_shape=(height, width, channels)\nimg_size=(height, width)\nlength=len(test_df)\ntest_batch_size=sorted([int(length/n) for n in range(1,length+1) if length % n ==0 and length/n<=80],reverse=True)[0] \ntest_steps=int(length/test_batch_size)\nprint ( 'test batch size: ' ,test_batch_size, ' test steps: ', test_steps)\ndef scalar(img):\n #img=img/127.5-1\n return img \ntrgen=ImageDataGenerator(preprocessing_function=scalar, horizontal_flip=True)\ntvgen=ImageDataGenerator(preprocessing_function=scalar)\ntrain_gen=trgen.flow_from_dataframe( train_df, x_col='filepaths', y_col='labels', target_size=img_size, class_mode='categorical',\n color_mode='rgb', shuffle=True, batch_size=batch_size)\ntest_gen=tvgen.flow_from_dataframe( test_df, x_col='filepaths', y_col='labels', target_size=img_size, class_mode='categorical',\n color_mode='rgb', shuffle=False, batch_size=test_batch_size)\nvalid_gen=tvgen.flow_from_dataframe( valid_df, x_col='filepaths', y_col='labels', target_size=img_size, class_mode='categorical',\n color_mode='rgb', shuffle=True, batch_size=batch_size)\nclasses=list(train_gen.class_indices.keys())\nclass_count=len(classes)\ntrain_steps=int(len(train_gen.labels)/batch_size)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-18T18:06:21.416126Z","iopub.execute_input":"2022-02-18T18:06:21.416489Z","iopub.status.idle":"2022-02-18T18:06:21.451532Z","shell.execute_reply.started":"2022-02-18T18:06:21.416445Z","shell.execute_reply":"2022-02-18T18:06:21.450604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Plan C","metadata":{}},{"cell_type":"code","source":"from PIL import Image\n\nimport keras\nfrom keras.layers import Conv2D,Flatten,Dense,MaxPooling2D\nfrom keras.preprocessing.image import ImageDataGenerator","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:06:28.081621Z","iopub.execute_input":"2022-02-18T18:06:28.081888Z","iopub.status.idle":"2022-02-18T18:06:28.087104Z","shell.execute_reply.started":"2022-02-18T18:06:28.081863Z","shell.execute_reply":"2022-02-18T18:06:28.086026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by Gajulamandyam Deva Kumar https://www.kaggle.com/devaloo/kerneledb31c4357\n\ntrain_datagen = ImageDataGenerator(rescale = 1/255)\ntest_datagen = ImageDataGenerator(rescale = 1/255)","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:06:32.705762Z","iopub.execute_input":"2022-02-18T18:06:32.70648Z","iopub.status.idle":"2022-02-18T18:06:32.712063Z","shell.execute_reply.started":"2022-02-18T18:06:32.706432Z","shell.execute_reply":"2022-02-18T18:06:32.711273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Class_mode (sparse/binary/categorical) I choose sparse\n\n\"The sparse is not used in flow_from_directory and usually used when an image can be more than one class. For example if an image can be A and C the target will be (1, 0, 1).\"\n\n\"The reason for binary actually called this way because at the earlier versions it only gives you 1 and 0.\"\n\nhttps://stackoverflow.com/questions/63867609/what-is-the-difference-between-binary-sparse-and-categorical-in-class-mode","metadata":{}},{"cell_type":"code","source":"trainset = train_datagen.flow_from_directory(r\"../input/herbarium-2022-fgvc9/train_images\",target_size = (150,150),class_mode='sparse',batch_size=80)\ntestset = test_datagen.flow_from_directory(r\"../input/herbarium-2022-fgvc9/test_images\",target_size = (150,150),class_mode = 'sparse',batch_size= 20)","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:06:38.179641Z","iopub.execute_input":"2022-02-18T18:06:38.180156Z","iopub.status.idle":"2022-02-18T18:07:49.002121Z","shell.execute_reply.started":"2022-02-18T18:06:38.180122Z","shell.execute_reply":"2022-02-18T18:07:49.001381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by Gajulamandyam Deva Kumar https://www.kaggle.com/devaloo/kerneledb31c4357\n\nmodel = keras.models.Sequential([Conv2D(16,(3,3),input_shape = (150,150,3),activation = 'relu'),\n MaxPooling2D(2,2),\n Conv2D(32,(3,3),activation = 'relu'),\n MaxPooling2D(2,2),\n Conv2D(64,(3,3),activation = 'relu'),\n MaxPooling2D(2,2),\n Flatten(),\n Dense(512,activation = 'relu'),\n Dense(1,activation = 'sigmoid')\n \n])","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:09:14.028558Z","iopub.execute_input":"2022-02-18T18:09:14.028977Z","iopub.status.idle":"2022-02-18T18:09:14.347618Z","shell.execute_reply.started":"2022-02-18T18:09:14.028945Z","shell.execute_reply":"2022-02-18T18:09:14.345891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"No Learner Left Behind. The Complexity of Teaching Multiple Learners
\n\n\nNo Learner Left Behind: On the Complexity of Teaching Multiple Learners Simultaneously\n\nAuthors: Xiaojin Zhu, Ji Liu, Manuel Lopes - DOI:10.24963/ijcai.2017/502\n\n\"The authors presented a theoretical study of algorithmic teaching in the setting where the teacher must use the same training set to teach multiple learners. This problem is a theoretical abstraction of the real-world classroom setting in which the teacher delivers the same lecture to academically diverse students.\"\n\n\"They defined a minimax teaching criterion to guarantee the performance of the worst learner in the class. The authors proved that the teaching dimension increases with class diversity in general. For the classes of conjugate Bayesian learners and linear regression learners, respectively, they exhibit corresponding minimax teaching set.\"\n\n\"Then they proposed a method to enhance teaching by partitioning the class into sections. The authors presented cases where the optimal partition minimizes overall teaching dimension while maintaining the guarantee on all learners. Interestingly, they show personalized education (one learner per section) is not necessarily the optimal partition. Our results generalize algorithmic teaching to multiple learners and offer insight on how to teach large classes.\"\n\nhttps://www.researchgate.net/publication/318830150_No_Learner_Left_Behind_On_the_Complexity_of_Teaching_Multiple_Learners_Simultaneously","metadata":{}},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:09:18.627675Z","iopub.execute_input":"2022-02-18T18:09:18.627934Z","iopub.status.idle":"2022-02-18T18:09:18.639813Z","shell.execute_reply.started":"2022-02-18T18:09:18.627905Z","shell.execute_reply":"2022-02-18T18:09:18.638819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Probably Approximately Correct (PAC) Learning and VC Theory
\n\nDiscussion of \"No Learner Left Behind: On the Complexity of Teaching Multiple Learners Simultaneously\"\n\nAuthors: Xiaojin Zhu, Ji Liu, Manuel Lopes - DOI:10.24963/ijcai.2017/502\n\n\n\"Readers familiar with Probably Approximately Correct (PAC) learning and VC theory (Vapnik–Chervonenkis theory) may wonder how their work relates to those framework. The class TD in Definition 4 provides a good contrast. The authors are controlling the error to be uniformly below. But a major difference is that we do not rely on concentration of measure over i.i.d. samples in the training set. Instead, we start with the target concept θ ∗ and then optimize the training set D.\"\n\n\"Therefore, we do not have the “P” in PAC learning. In terms of VC dimension, it is known that VC and TD are distinct measures, and the distinction carries over to our classroom teaching setting. In order to make analysis tractable, we made several sim\u0002plifying assumptions that should be relaxed in future work. A strong assumption is that the teacher knows everything about the students A (i.e., their learning algorithms). Future work may allow the teacher to probe the students if this is not the case, and jointly optimize the effort of teaching and probing.\n\n\"A variant that is ethically questionable but unfortunately occurs in practice is for a teacher to give up on at most δ frac\u0002tion of the students, to allow them to have arbitrarily large stu\u0002dent failures. The remaining students must -approximately learn. This type of (, δ) class TD can be much smaller. Interestingly, in the education literature it was suggested that some cultures prefer large classes, with the argument that students can observe and learn from each other’s mistakes. \n\n\"It is possible to build this into machine teaching theory by formulating the class A as a multi-agent system. More research is needed to understand how academic diversity benefits teaching.\"\n\n\"In summary, this paper advances machine teaching theory and provides a potential theoretical framework for optimizing\nreal-world education resource allocation in the future.\"\n\nhttps://www.ijcai.org/proceedings/2017/0502.pdf","metadata":{}},{"cell_type":"code","source":"tf.keras.losses.SparseCategoricalCrossentropy(\n from_logits=False, reduction=\"auto\", name=\"sparse_categorical_crossentropy\"\n)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-18T18:09:24.268739Z","iopub.execute_input":"2022-02-18T18:09:24.269112Z","iopub.status.idle":"2022-02-18T18:09:24.275253Z","shell.execute_reply.started":"2022-02-18T18:09:24.269087Z","shell.execute_reply":"2022-02-18T18:09:24.274166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#from tensorflow.keras.optimizers import RMSprop\n\n#model.compile(optimizer = RMSprop(learning_rate = 0.001),loss = 'SparseCategoricalCrossentropy',metrics = ['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:09:28.994785Z","iopub.execute_input":"2022-02-18T18:09:28.995055Z","iopub.status.idle":"2022-02-18T18:09:28.999264Z","shell.execute_reply.started":"2022-02-18T18:09:28.995028Z","shell.execute_reply":"2022-02-18T18:09:28.998517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.optimizers import RMSprop\n\nmodel.compile(optimizer = RMSprop(learning_rate = 0.001),loss = 'binary_crossentropy',metrics = ['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:09:33.312272Z","iopub.execute_input":"2022-02-18T18:09:33.312562Z","iopub.status.idle":"2022-02-18T18:09:33.327763Z","shell.execute_reply.started":"2022-02-18T18:09:33.312511Z","shell.execute_reply":"2022-02-18T18:09:33.326729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It isn't Binary!!! Though when I changed loss/metrics Plan C didn't work too.","metadata":{}},{"cell_type":"code","source":"#https://keras.io/api/losses/probabilistic_losses/#sparsecategoricalcrossentropy-class\n\n#Didn't work to fit the model\n\n#model.compile(optimizer='sgd',\n #loss=tf.keras.losses.SparseCategoricalCrossentropy())","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:09:39.458498Z","iopub.execute_input":"2022-02-18T18:09:39.45896Z","iopub.status.idle":"2022-02-18T18:09:39.462825Z","shell.execute_reply.started":"2022-02-18T18:09:39.458923Z","shell.execute_reply":"2022-02-18T18:09:39.461609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Competition Metric is: macro F1 score.","metadata":{}},{"cell_type":"markdown","source":"#Loss: binary_crossentropy, SparseCategoricalCrossentropy,CategoricalCrossentropy","metadata":{}},{"cell_type":"code","source":"model.fit(trainset,steps_per_epoch=50,epochs = 10,validation_data=testset,validation_steps=20)","metadata":{"execution":{"iopub.status.busy":"2022-02-18T18:09:44.329403Z","iopub.execute_input":"2022-02-18T18:09:44.32968Z","iopub.status.idle":"2022-02-18T18:31:05.942312Z","shell.execute_reply.started":"2022-02-18T18:09:44.329654Z","shell.execute_reply":"2022-02-18T18:31:05.940749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#It's not a Binary, I'm in the loop. Not the model.","metadata":{}},{"cell_type":"code","source":"trainset.class_indices","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-18T18:31:56.763399Z","iopub.execute_input":"2022-02-18T18:31:56.763764Z","iopub.status.idle":"2022-02-18T18:31:56.779026Z","shell.execute_reply.started":"2022-02-18T18:31:56.763732Z","shell.execute_reply":"2022-02-18T18:31:56.777933Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from PIL import Image\n\nimage = Image.open(\"../input/herbarium-2022-fgvc9/train_images/002/02/00202__002.jpg\")\nimage","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-18T18:33:09.682288Z","iopub.execute_input":"2022-02-18T18:33:09.68266Z","iopub.status.idle":"2022-02-18T18:33:09.86158Z","shell.execute_reply.started":"2022-02-18T18:33:09.682635Z","shell.execute_reply":"2022-02-18T18:33:09.860084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Code by Gajulamandyam Deva Kumar https://www.kaggle.com/devaloo/kerneledb31c4357\n\nfrom keras.preprocessing import image\ntestdata1 = image.load_img(\"../input/herbarium-2022-fgvc9/train_images/002/02/00202__002.jpg\",target_size = (150,150))\nimport numpy as np\nx1=image.img_to_array(testdata1)\nx1=np.expand_dims(x1, axis=0)\nimages = np.vstack([x1])\nans = model.predict(images)\nif ans[0][0] == 0:\n print(\"It is a Plant\")\nelse:\n print(\"It is Definitely a Plant\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-18T18:33:01.788434Z","iopub.execute_input":"2022-02-18T18:33:01.788721Z","iopub.status.idle":"2022-02-18T18:33:02.006481Z","shell.execute_reply.started":"2022-02-18T18:33:01.788695Z","shell.execute_reply":"2022-02-18T18:33:02.00578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#\"The authors defined a minimax teaching criterion to guarantee the performance of the worst learner in the class.\"\n\nAfter the presentation above, very likely that clueless person it's me.","metadata":{}},{"cell_type":"markdown","source":"Acknowledgements:\n\nDatalira https://www.kaggle.com/databeru/classify-bricks-compare-transfer-learning-model/notebook\n\nGerry https://www.kaggle.com/gpiosenka/gym-f1-score-100/data?select=MontezumaRevenge\n\nGajulamandyam Deva Kumar https://www.kaggle.com/devaloo/kerneledb31c4357","metadata":{}},{"cell_type":"markdown","source":"hai.stanford.edu","metadata":{}}]}