{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n# Introduction #\n\nKaggleにはCompetition(大会)があり主に人工知能や統計分析に関する大会が開かれています．\n\nこのコードは花の分類器を実装する大会用のコードで，実際に出場したものです．\n\n約93%の精度で100種類以上の花を分類することができます．\n\nこのコードには**学習済みのモデル**を既にダウンロードしてあります．\n\n\n\n## 実行する前に右上の3点リーダ→Acceleratorから**TPU**をオンにしてください\n\n","metadata":{}},{"cell_type":"markdown","source":"# Step 1: Imports #\nWe begin by importing several Python packages.\n\nパッケージのインポート,tensorflowのバージョン確認","metadata":{}},{"cell_type":"code","source":"import math, re, os\nimport numpy as np\nimport tensorflow as tf\n\n# バージョン確認\nprint(\"Tensorflow version \" + tf.__version__)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T05:44:04.934894Z","iopub.execute_input":"2021-12-22T05:44:04.935809Z","iopub.status.idle":"2021-12-22T05:44:10.090255Z","shell.execute_reply.started":"2021-12-22T05:44:04.935682Z","shell.execute_reply":"2021-12-22T05:44:10.089658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 2: Distribution Strategy #\n\nA TPU has eight different *cores* and each of these cores acts as its own accelerator. (A TPU is sort of like having eight GPUs in one machine.) We tell TensorFlow how to make use of all these cores at once through a **distribution strategy**. Run the following cell to create the distribution strategy that we'll later apply to our model.\n\nTPUの設定,(TPUは8個のコアを持つが，それを全て使うための設定)","metadata":{}},{"cell_type":"code","source":"# Detect TPU, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T05:44:10.092478Z","iopub.execute_input":"2021-12-22T05:44:10.093360Z","iopub.status.idle":"2021-12-22T05:44:15.648512Z","shell.execute_reply.started":"2021-12-22T05:44:10.093316Z","shell.execute_reply":"2021-12-22T05:44:15.647765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll use the distribution strategy when we create our neural network model. Then, TensorFlow will distribute the training among the eight TPU cores by creating eight different *replicas* of the model, one for each core.\n\nこの設定を入れることで，一つのTPUで8つのGPUを使う\n\n# Step 3: Loading the Competition Data #\n\n## Get GCS Path ##\n\nWhen used with TPUs, datasets need to be stored in a [Google Cloud Storage bucket](https://cloud.google.com/storage/). You can use data from any public GCS bucket by giving its path just like you would data from `'/kaggle/input'`. The following will retrieve the GCS path for this competition's dataset.\n\nデータセットのPATHをとる","metadata":{}},{"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started')\nprint(GCS_DS_PATH) # Google Cloud Strage上のデータセットのPATH","metadata":{"execution":{"iopub.status.busy":"2021-12-22T05:44:15.649624Z","iopub.execute_input":"2021-12-22T05:44:15.650188Z","iopub.status.idle":"2021-12-22T05:44:16.130614Z","shell.execute_reply.started":"2021-12-22T05:44:15.650148Z","shell.execute_reply":"2021-12-22T05:44:16.129977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can use data from any public dataset here on Kaggle in just the same way. If you'd like to use data from one of your private datasets, see [here](https://www.kaggle.com/docs/tpu#tpu3pt5).\n\n## Load Data ##\n\nWhen used with TPUs, datasets are often serialized into [TFRecords](https://www.kaggle.com/ryanholbrook/tfrecords-basics). This is a format convenient for distributing data to each of the TPUs cores. We've hidden the cell that reads the TFRecords for our dataset since the process is a bit long. You could come back to it later for some guidance on using your own datasets with TPUs.\n\nデータセットは複数の画像であるが,jpegやpng形式ではなくTFRecordsという形式のバイナリデータである．\n\nこれはTPUで学習する際扱いやすいためのデータである．","metadata":{}},{"cell_type":"code","source":"IMAGE_SIZE = [512, 512] # 512*512 pixel(元データのサイズ)\n\n\n\n# Subdirs by image size\nGCS_PATH_SELECT = { # available image sizes\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\n# Select the dataset containing the size we chose above\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\nAUTO = tf.data.experimental.AUTOTUNE # CPUとTPUで並列処理をする時のためのもの\n\n# 訓練用データ,評価用データ，テスト用データのPATH指定\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec')\n\n\n# 花の種類\nCLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102\n\n# 入力画像をjpeg形式にデコードして3次元のテンソルにして返す関数\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)  #RGBなのでchannel 3\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\n# 画像のラベル付けを行う関数\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\n# テストデータ用にラベルを外す関数\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\n# データセットを読み込む関数\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n    \n    # とにかく順番を無視して並列処理でデータを読み込むスピードを上げる．(後でデータはシャッフルするので)\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-22T05:44:16.132346Z","iopub.execute_input":"2021-12-22T05:44:16.132605Z","iopub.status.idle":"2021-12-22T05:44:16.381456Z","shell.execute_reply.started":"2021-12-22T05:44:16.132578Z","shell.execute_reply":"2021-12-22T05:44:16.380611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create Data Pipelines ##\n\nIn this final step we'll use the `tf.data` API to define an efficient data pipeline for each of the training, validation, and test splits.\n\ntensorflowのAPIを使って実際に学習データと評価データとテストデータを読み込む","metadata":{}},{"cell_type":"code","source":"# 学習データを増やす,左から右へ反転したデータ(鏡像?)\ndef data_augment(image, label):\n    # Thanks to the dataset.prefetch(AUTO)\n    # statement in the next function (below), this happens essentially\n    # for free on TPU. Data pipeline code is executed on the \"CPU\"\n    # part of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(image)\n    #image = tf.image.random_saturation(image, 0, 2)\n    return image, label   \n\n# 学習用データを取り込む関数\ndef get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    # 訓練用データにaugmentデータを追加\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    ### old code\n    ### dataset = dataset.shuffle(2048)\n    dataset = dataset.shuffle(3712)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\n# 評価用データを取り込む関数\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\n# テスト用データを取り込む関数\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\n# .tfrec形式だと何枚の画像かわからないので数える関数\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec\n    # files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\n# 数えさせる\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-22T05:44:16.382614Z","iopub.execute_input":"2021-12-22T05:44:16.383124Z","iopub.status.idle":"2021-12-22T05:44:16.397724Z","shell.execute_reply.started":"2021-12-22T05:44:16.383077Z","shell.execute_reply":"2021-12-22T05:44:16.396645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This next cell will create the datasets that we'll use with Keras during training and inference. Notice how we scale the size of the batches to the number of TPU cores.\n\n実際に関数を使ってds_train,ds_valid,ds_testにそれぞれデータを入力する．","metadata":{}},{"cell_type":"code","source":"# バッチサイズをTPUに合わせて定義\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\n# 学習用データ，評価用データ，テスト用データを取り込む関数を実行\nds_train = get_training_dataset()\nds_valid = get_validation_dataset()\nds_test = get_test_dataset()\n\nprint(\"Training:\", ds_train)\nprint (\"Validation:\", ds_valid)\nprint(\"Test:\", ds_test)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T05:44:16.398820Z","iopub.execute_input":"2021-12-22T05:44:16.399064Z","iopub.status.idle":"2021-12-22T05:44:16.723990Z","shell.execute_reply.started":"2021-12-22T05:44:16.399040Z","shell.execute_reply":"2021-12-22T05:44:16.723153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These datasets are `tf.data.Dataset` objects. You can think about a dataset in TensorFlow as a *stream* of data records. The training and validation sets are streams of `(image, label)` pairs.\n\n試しに学習用データから3枚画像とりだしてshapeとlabelが適当であるか確かめる","metadata":{}},{"cell_type":"code","source":"np.set_printoptions(threshold=15, linewidth=80)\n\nprint(\"Training data shapes:\")\nfor image, label in ds_train.take(3):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Training data label examples:\", label.numpy())","metadata":{"execution":{"iopub.status.busy":"2021-12-22T05:44:16.725377Z","iopub.execute_input":"2021-12-22T05:44:16.725867Z","iopub.status.idle":"2021-12-22T05:44:22.409967Z","shell.execute_reply.started":"2021-12-22T05:44:16.725835Z","shell.execute_reply":"2021-12-22T05:44:22.408544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The test set is a stream of `(image, idnum)` pairs; `idnum` here is the unique identifier given to the image．\n\nテスト用データはlabelはないので画像固有のIDが割り振られていることを確認．","metadata":{}},{"cell_type":"code","source":"print(\"Test data shapes:\")\nfor image, idnum in ds_test.take(3):\n    print(image.numpy().shape, idnum.numpy().shape)\nprint(\"Test data IDs:\", idnum.numpy().astype('U')) # U=unicode string","metadata":{"execution":{"iopub.status.busy":"2021-12-22T05:44:22.411127Z","iopub.execute_input":"2021-12-22T05:44:22.411673Z","iopub.status.idle":"2021-12-22T05:44:25.706136Z","shell.execute_reply.started":"2021-12-22T05:44:22.411621Z","shell.execute_reply":"2021-12-22T05:44:25.704687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 4: Explore Data #\n\nLet's take a moment to look at some of the images in the dataset.\n\nデータセットの中身を実際に見てみる","metadata":{}},{"cell_type":"code","source":"\nfrom matplotlib import pyplot as plt\n\n# ラベルがない場合は固有のIDを振る\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case,\n                                     # these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is\n    # the case for test data)\n    return numpy_images, numpy_labels\n\n# ラベルと花の名前を変換する関数\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\n# 1枚の花の画像を表示する関数\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n\n# 画像を表示\ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square\n    # or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.savefig(\"sample2.png\")\n    plt.show()\n\n# 学習レートの表示(今回は学習済みモデルなので実行しない)\ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-22T05:44:25.708366Z","iopub.execute_input":"2021-12-22T05:44:25.708833Z","iopub.status.idle":"2021-12-22T05:44:25.731611Z","shell.execute_reply.started":"2021-12-22T05:44:25.708788Z","shell.execute_reply":"2021-12-22T05:44:25.730779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can display a single batch of images from a dataset with another of our helper functions. The next cell will turn the dataset into an iterator of batches of 20 images.","metadata":{}},{"cell_type":"code","source":"ds_iter = iter(ds_train.unbatch().batch(20))","metadata":{"execution":{"iopub.status.busy":"2021-12-22T05:44:25.733980Z","iopub.execute_input":"2021-12-22T05:44:25.734509Z","iopub.status.idle":"2021-12-22T05:44:25.754285Z","shell.execute_reply.started":"2021-12-22T05:44:25.734480Z","shell.execute_reply":"2021-12-22T05:44:25.753495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Use the Python `next` function to pop out the next batch in the stream and display it with the helper function.\n\nランダムに花の画像を20枚表示してみる","metadata":{}},{"cell_type":"code","source":"one_batch = next(ds_iter)\ndisplay_batch_of_images(one_batch)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T05:44:25.755153Z","iopub.execute_input":"2021-12-22T05:44:25.755650Z","iopub.status.idle":"2021-12-22T05:44:32.709051Z","shell.execute_reply.started":"2021-12-22T05:44:25.755622Z","shell.execute_reply":"2021-12-22T05:44:32.707061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By defining `ds_iter` and `one_batch` in separate cells, you only need to rerun the cell above to see a new batch of images.","metadata":{}},{"cell_type":"markdown","source":"# Step 5,6:Import Models\n","metadata":{}},{"cell_type":"markdown","source":"学習済みモデルをダウンロード","metadata":{}},{"cell_type":"code","source":"from tensorflow import keras \nmodel = keras.models.load_model(\"../input/flower-classification-model/flower-classification.h5\")\n","metadata":{"execution":{"iopub.status.busy":"2021-12-22T05:44:32.710204Z","iopub.execute_input":"2021-12-22T05:44:32.710429Z","iopub.status.idle":"2021-12-22T05:44:50.449488Z","shell.execute_reply.started":"2021-12-22T05:44:32.710402Z","shell.execute_reply":"2021-12-22T05:44:50.448474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 7: Evaluate Predictions #\n\nBefore making your final predictions on the test set, it's a good idea to evaluate your model's predictions on the validation set. This can help you diagnose problems in training or suggest ways your model could be improved. We'll look at two common ways of validation: plotting the **confusion matrix** and **visual validation**.\n\n評価用データを入力し，モデルが正しく分類できていることを確認する．","metadata":{}},{"cell_type":"code","source":"\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\n\n# 評価用データとモデルが予測したものをそれぞれ色付けしたmatrixを表示\ndef display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()\n    \ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-22T05:44:50.456458Z","iopub.execute_input":"2021-12-22T05:44:50.456746Z","iopub.status.idle":"2021-12-22T05:44:51.132682Z","shell.execute_reply.started":"2021-12-22T05:44:50.456713Z","shell.execute_reply":"2021-12-22T05:44:51.131844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Confusion Matrix ##\n\nA [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) shows the actual class of an image tabulated against its predicted class. It is one of the best tools you have for evaluating the performance of a classifier.\n\nThe following cell does some processing on the validation data and then creates the matrix with the `confusion_matrix` function included in [`scikit-learn`](https://scikit-learn.org/stable/index.html).\n\n実際にモデルの予測と正解がすべてmatrixになったものを表示してみる","metadata":{}},{"cell_type":"code","source":"cmdataset = get_validation_dataset(ordered=True)\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\n\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy()\ncm_probabilities = model.predict(images_ds)\ncm_predictions = np.argmax(cm_probabilities, axis=-1)\n\nlabels = range(len(CLASSES))\ncmat = confusion_matrix(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n)\ncmat = (cmat.T / cmat.sum(axis=1)).T # normalize","metadata":{"execution":{"iopub.status.busy":"2021-12-22T05:44:51.134005Z","iopub.execute_input":"2021-12-22T05:44:51.134425Z","iopub.status.idle":"2021-12-22T05:48:56.731312Z","shell.execute_reply.started":"2021-12-22T05:44:51.134381Z","shell.execute_reply":"2021-12-22T05:48:56.730430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You might be familiar with metrics like [F1-score](https://en.wikipedia.org/wiki/F1_score) or [precision and recall](https://en.wikipedia.org/wiki/Precision_and_recall). This cell will compute these metrics and display them with a plot of the confusion matrix. (These metrics are defined in the Scikit-learn module `sklearn.metrics`; we've imported them in the helper script for you.)\n\nF1-scoreと呼ばれる正答率を基準にモデルを評価する．","metadata":{}},{"cell_type":"code","source":"score = f1_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\nprecision = precision_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\nrecall = recall_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\ndisplay_confusion_matrix(cmat, score, precision, recall)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T05:48:56.732669Z","iopub.execute_input":"2021-12-22T05:48:56.732978Z","iopub.status.idle":"2021-12-22T05:49:01.750479Z","shell.execute_reply.started":"2021-12-22T05:48:56.732949Z","shell.execute_reply":"2021-12-22T05:49:01.749620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visual Validation ##\n\nIt can also be helpful to look at some examples from the validation set and see what class your model predicted. This can help reveal patterns in the kinds of images your model has trouble with.\n\nThis cell will set up the validation set to display 20 images at a time -- you can change this to display more or fewer, if you like.\n\n評価を視覚化してみる．実際の画像と，モデルが予測した品種，またそれが合っているのか間違っているのかを見やすく表示してみる","metadata":{}},{"cell_type":"code","source":"dataset = get_validation_dataset()\ndataset = dataset.unbatch().batch(20)\nbatch = iter(dataset)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T05:49:01.751714Z","iopub.execute_input":"2021-12-22T05:49:01.751939Z","iopub.status.idle":"2021-12-22T05:49:01.791577Z","shell.execute_reply.started":"2021-12-22T05:49:01.751913Z","shell.execute_reply":"2021-12-22T05:49:01.790813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images, labels = next(batch)\nprobabilities = model.predict(images)\npredictions = np.argmax(probabilities, axis=-1)\ndisplay_batch_of_images((images, labels), predictions)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T05:49:01.792584Z","iopub.execute_input":"2021-12-22T05:49:01.792801Z","iopub.status.idle":"2021-12-22T05:49:09.002556Z","shell.execute_reply.started":"2021-12-22T05:49:01.792776Z","shell.execute_reply":"2021-12-22T05:49:09.001720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161321) to chat with other Learners.*","metadata":{}}]}