{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip install -q efficientnet\n\nimport gc\nimport warnings\nimport sklearn.exceptions\nwarnings.filterwarnings('ignore', category=DeprecationWarning)\nwarnings.filterwarnings('ignore', category=FutureWarning)\nwarnings.filterwarnings(\"ignore\", category=sklearn.exceptions.UndefinedMetricWarning)\n\nimport os\nimport re\nimport numpy as np\n\nimport tensorflow as tf\nimport tensorflow_addons as tfa\nimport efficientnet.tfkeras as efn\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\n\nfrom IPython.display import Image, display\nfrom matplotlib.cm import ScalarMappable\nfrom tensorflow.keras.models import Model, load_model\nfrom tensorflow.keras.layers import Input, Dense, Layer, InputSpec\nfrom tensorflow.keras.callbacks import ModelCheckpoint, TensorBoard\nfrom tensorflow.keras import regularizers, activations, initializers, constraints, Sequential\nfrom tensorflow.keras import backend as K\nfrom tensorflow.keras.constraints import UnitNorm, Constraint\n\nfrom tqdm.notebook import tqdm\nfrom kaggle_datasets import KaggleDatasets\nfrom glob import glob","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:13:35.319405Z","iopub.execute_input":"2021-07-07T13:13:35.319772Z","iopub.status.idle":"2021-07-07T13:13:53.625943Z","shell.execute_reply.started":"2021-07-07T13:13:35.319743Z","shell.execute_reply":"2021-07-07T13:13:53.624689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"\n

Table of Contents

","metadata":{"execution":{"iopub.status.busy":"2021-07-07T14:43:42.241567Z","iopub.execute_input":"2021-07-07T14:43:42.241969Z","iopub.status.idle":"2021-07-07T14:43:42.250315Z","shell.execute_reply.started":"2021-07-07T14:43:42.241926Z","shell.execute_reply":"2021-07-07T14:43:42.248683Z"}}},{"cell_type":"markdown","source":"1. [Load Data](#1)\n2. [AutoEncoder](#2) \n 2.1 [Utils](#2.1) \n 2.2 [Train AutoEncoder](#2.2) \n 2.3 [Freezed Pretrained EfficientNetB1 with Encoder](#2.3)\n3. [Train dataset analysis](#3) \n 3.1 [Group 1: Without gravitation wave](#3.1) \n 3.2 [Group 2: With GW & Probability > 0.8 ](#3.2) \n 3.3 [With GW & Probability < 0.8](#3.3)\n4. [Train/Test dataset comparission](#4) \n5. [Grad-CAM](#5)\n6. [Conclusion](#6)\n7. [References](#7)","metadata":{}},{"cell_type":"markdown","source":"\n

1. Load Data
","metadata":{}},{"cell_type":"markdown","source":"Train Datasets\n* [Q-Transform TFRecords](https://www.kaggle.com/miklgr500/q-transform-tfrecords)\n * [CQT G2Net V2 [0 - 1]](https://www.kaggle.com/miklgr500/cqt-g2net-v2-0-1)\n * [CQT G2Net V2 [2 - 3]](https://www.kaggle.com/miklgr500/cqt-g2net-v2-2-3)\n * [CQT G2Net V2 [4 - 5]](https://www.kaggle.com/miklgr500/cqt-g2net-v2-4-5)\n * [CQT G2Net V2 [6 - 7]](https://www.kaggle.com/miklgr500/cqt-g2net-v2-6-7)\n * [CQT G2Net V2 [8 - 9]](https://www.kaggle.com/miklgr500/cqt-g2net-v2-8-9)\n * [CQT G2Net V2 [10 - 11]](https://www.kaggle.com/miklgr500/cqt-g2net-v2-10-11)\n * [CQT G2Net V2 [12 - 13]](https://www.kaggle.com/miklgr500/cqt-g2net-v2-12-13)\n * [CQT G2Net V2 [14 - 15]](https://www.kaggle.com/miklgr500/cqt-g2net-v2-14-15)\n \nTest Datasets\n* [CQT G2Net Test [0 - 1]](https://www.kaggle.com/miklgr500/cqt-g2net-test-0-1)\n* [CQT G2Net Test [2 - 3]](https://www.kaggle.com/miklgr500/cqt-g2net-test-2-3)\n* [CQT G2Net Test [4 - 5]](https://www.kaggle.com/miklgr500/cqt-g2net-test-4-5)\n* [CQT G2Net Test [6 - 7]](https://www.kaggle.com/miklgr500/cqt-g2net-test-6-7)","metadata":{}},{"cell_type":"code","source":"IMG_SIZES = 256\nBATCH_SIZE = 32\nEPOCHS = 256\n\nEPOCHS_VIS = 500\n\n# From https://www.kaggle.com/xhlulu/ranzcr-efficientnet-tpu-training\ndef auto_select_accelerator():\n TPU_DETECTED = False\n try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\n print(\"Running on TPU:\", tpu.master())\n TPU_DETECTED =True\n except ValueError:\n strategy = tf.distribute.get_strategy()\n print(f\"Running on {strategy.num_replicas_in_sync} replicas\")\n \n return strategy, TPU_DETECTED\n\n\nstrategy, TPU_DETECTED = auto_select_accelerator()\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:13:53.627807Z","iopub.execute_input":"2021-07-07T13:13:53.628319Z","iopub.status.idle":"2021-07-07T13:13:53.645326Z","shell.execute_reply.started":"2021-07-07T13:13:53.62827Z","shell.execute_reply":"2021-07-07T13:13:53.643995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"files_train_g = []\nfor i,k in tqdm([(0, 1), (2, 3), (4,5), (6, 7), (8, 9) ,(10,11), (12, 13), (14, 15)]):\n GCS_PATH = KaggleDatasets().get_gcs_path(f'cqt-g2net-v2-{i}-{k}')\n files_train_g.extend(np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/train*.tfrec'))).tolist())\nnum_train_files = len(files_train_g)\nprint('train_files:',num_train_files)\n\nfiles_test_g = []\nfor i,k in tqdm([(0, 1), (2, 3), (4, 5), (6, 7)]):\n GCS_PATH = KaggleDatasets().get_gcs_path(f'cqt-g2net-test-{i}-{k}')\n files_test_g.extend(np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/test*.tfrec'))).tolist())\nnum_train_files = len(files_test_g)\nprint('test_files:',num_train_files)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:13:53.648361Z","iopub.execute_input":"2021-07-07T13:13:53.64938Z","iopub.status.idle":"2021-07-07T13:14:02.859264Z","shell.execute_reply.started":"2021-07-07T13:13:53.649334Z","shell.execute_reply":"2021-07-07T13:14:02.858031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_labeled_tfrecord(example):\n tfrec_format = {\n 'image' : tf.io.FixedLenFeature([], tf.string),\n 'image_id' : tf.io.FixedLenFeature([], tf.string),\n 'target' : tf.io.FixedLenFeature([], tf.int64)\n } \n example = tf.io.parse_single_example(example, tfrec_format)\n return prepare_image(example['image']), tf.reshape(tf.cast(example['target'], tf.float32), [1])\n\n\ndef read_unlabeled_tfrecord(example, return_image_id):\n tfrec_format = {\n 'image' : tf.io.FixedLenFeature([], tf.string),\n 'image_id' : tf.io.FixedLenFeature([], tf.string),\n }\n example = tf.io.parse_single_example(example, tfrec_format)\n return prepare_image(example['image']), example['image_id'] if return_image_id else 0\n\n \ndef prepare_image(img, dim=IMG_SIZES): \n img = tf.image.resize(tf.image.decode_png(img, channels=3), size=(dim, dim))\n img = tf.cast(img, tf.float32) / 255.0\n img = tf.reshape(img, [dim,dim, 3])\n \n return img\n\ndef count_data_items(fileids):\n n = [int(re.compile(r\"-([0-9]*)\\.\").search(fileid).group(1)) \n for fileid in fileids]\n return np.sum(n)\n\n\ndef get_dataset(files, shuffle = False, repeat = False, \n labeled=True, return_image_ids=True, batch_size=BATCH_SIZE, dim=IMG_SIZES):\n \n ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n #ds = ds.cache()\n \n if repeat:\n ds = ds.repeat()\n \n if shuffle: \n ds = ds.shuffle(1024*2)\n opt = tf.data.Options()\n opt.experimental_deterministic = False\n ds = ds.with_options(opt)\n \n if labeled: \n ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n else:\n ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_ids), \n num_parallel_calls=AUTO) \n\n ds = ds.batch(batch_size * REPLICAS)\n ds = ds.prefetch(AUTO)\n return ds","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:14:02.861428Z","iopub.execute_input":"2021-07-07T13:14:02.862205Z","iopub.status.idle":"2021-07-07T13:14:02.879091Z","shell.execute_reply.started":"2021-07-07T13:14:02.862144Z","shell.execute_reply":"2021-07-07T13:14:02.877825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

2. AutoEncoder
","metadata":{}},{"cell_type":"markdown","source":"For 2D/3D image data visualization need to reduce dimensions with saving useful information. And exist a number of techniques for doing this, one of which is autoencoder.\n\nPretrained model:\n* [CQT G2Net EfficientNetB1[TPU Training]](https://www.kaggle.com/miklgr500/cqt-g2net-efficientnetb1-tpu-training)\n* [CQT G2Net EfficientNetB1[TPU Inference] ](https://www.kaggle.com/miklgr500/cqt-g2net-efficientnetb1-tpu-inference)\n\n\"Untitled-Diagram-1\"","metadata":{}},{"cell_type":"markdown","source":"\n

2.1 Utils
","metadata":{}},{"cell_type":"code","source":"class UncorrelatedFeaturesConstraint (Constraint):\n def __init__(self, encoding_dim, weightage = 1.0):\n self.encoding_dim = encoding_dim\n self.weightage = weightage\n \n def get_covariance(self, x):\n x_centered_list = []\n\n for i in range(self.encoding_dim):\n x_centered_list.append(x[:, i] - K.mean(x[:, i]))\n \n x_centered = tf.stack(x_centered_list)\n covariance = K.dot(x_centered, K.transpose(x_centered)) / tf.cast(x_centered.get_shape()[0], tf.float32)\n \n return covariance\n \n # Constraint penalty\n def uncorrelated_feature(self, x, covariance):\n if(self.encoding_dim <= 1):\n return 0.0\n else:\n output = K.sum(K.square(\n covariance - tf.linalg.band_part(covariance, 0, 0)))\n return output\n\n def __call__(self, x):\n covariance = self.get_covariance(x)\n return self.weightage * self.uncorrelated_feature(x, covariance)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:14:02.880981Z","iopub.execute_input":"2021-07-07T13:14:02.881491Z","iopub.status.idle":"2021-07-07T13:14:02.896695Z","shell.execute_reply.started":"2021-07-07T13:14:02.881445Z","shell.execute_reply":"2021-07-07T13:14:02.895458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_model(size):\n inp = tf.keras.layers.Input(shape=(size, size,3))\n base = efn.EfficientNetB1(input_shape=(size,size,3),weights='imagenet',include_top=False)\n \n x = base(inp)\n \n x = tf.keras.layers.GlobalAvgPool2D()(x)\n \n x = tf.keras.layers.Dropout(0.2)(x)\n x = tf.keras.layers.Dense(1,activation='sigmoid')(x)\n model = tf.keras.Model(inputs=inp, outputs=x)\n opt = tf.optimizers.SGD(learning_rate=1e-3)\n loss = tf.keras.losses.BinaryCrossentropy() \n model.compile(optimizer=opt,loss=loss,metrics=['AUC'])\n return model\n\ndef build_model_gradcam(size):\n base = efn.EfficientNetB1(input_shape=(size,size,3),weights='imagenet',include_top=False)\n \n x = tf.keras.layers.GlobalAvgPool2D()(base.output)\n \n x = tf.keras.layers.Dropout(0.2)(x)\n x = tf.keras.layers.Dense(1,activation='sigmoid')(x)\n model = tf.keras.Model(inputs=base.input, outputs=x)\n opt = tf.optimizers.SGD(learning_rate=1e-3)\n loss = tf.keras.losses.BinaryCrossentropy() \n model.compile(optimizer=opt,loss=loss,metrics=['AUC'])\n return model\n\ndef get_emb_model(size, model):\n inp = tf.keras.layers.Input(shape=(size, size,3))\n base = efn.EfficientNetB1(input_shape=(size,size,3),weights='imagenet',include_top=False)\n \n x = base(inp)\n \n x = tf.keras.layers.GlobalAvgPool2D()(x)\n model = tf.keras.Model(inputs=inp, outputs=x)\n opt = tf.keras.optimizers.SGD(learning_rate=1e-3)\n loss = tf.keras.losses.BinaryCrossentropy() \n model.compile(optimizer=opt,loss=loss,metrics=['AUC'])\n return model \n\n\ndef get_ae_model(size, model):\n inp = tf.keras.layers.Input(shape=(size, size,3))\n emb = model(inp)\n encoder = tf.keras.layers.Dense(2, kernel_constraint=tf.keras.constraints.UnitNorm(axis=0),\n activity_regularizer=UncorrelatedFeaturesConstraint(2, weightage = 0.1))\n x = encoder(emb)\n x = tf.keras.layers.Dense(1280)(x)\n model = tf.keras.Model(inputs=inp, outputs=[x, emb])\n opt = tfa.optimizers.NovoGrad(learning_rate=1e-3)\n model.compile(optimizer=opt,loss='mse',metrics=['mse'])\n return model \n\ndef ae_model_to_e_model(size, base, ae_model):\n inp = tf.keras.layers.Input(shape=(size, size,3))\n emb = base(inp)\n x = tf.keras.layers.Dense(2, kernel_constraint=tf.keras.constraints.UnitNorm(axis=0),\n activity_regularizer=UncorrelatedFeaturesConstraint(2, weightage = 0.1))(emb)\n model = tf.keras.Model(inputs=inp, outputs=x)\n opt = tfa.optimizers.NovoGrad(learning_rate=1e-3)\n model.compile(optimizer=opt,loss='mse',metrics=['mse'])\n \n model.layers[-1].set_weights(ae_model.layers[-2].get_weights())\n return model \n\nmodel = build_model(256)\nmodel.load_weights('../input/cqt-g2net-efficientnetb1-tpu-training/fold-0.h5')\n\nemb_model = get_emb_model(256, model)\nemb_model.set_weights(model.get_weights()[:-2])\nfor l in emb_model.layers:\n l.trainable = False","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-07T13:15:38.485812Z","iopub.execute_input":"2021-07-07T13:15:38.486276Z","iopub.status.idle":"2021-07-07T13:15:48.461928Z","shell.execute_reply.started":"2021-07-07T13:15:38.486243Z","shell.execute_reply":"2021-07-07T13:15:48.460802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Freezed Pretrained EfficientNetB1","metadata":{}},{"cell_type":"code","source":"emb_model.summary()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:15:48.463829Z","iopub.execute_input":"2021-07-07T13:15:48.46433Z","iopub.status.idle":"2021-07-07T13:15:48.495707Z","shell.execute_reply.started":"2021-07-07T13:15:48.464287Z","shell.execute_reply":"2021-07-07T13:15:48.494301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

2.2. Train AutoEncoder
","metadata":{}},{"cell_type":"code","source":"train_ds = get_dataset(files_train_g, shuffle = True, repeat = False, labeled=True, return_image_ids=False)\n\nae_model = get_ae_model(256, emb_model)\noptimizer = tfa.optimizers.NovoGrad(learning_rate=1e-3)\n\nloss_values = []\nfor i, (x, y) in tqdm(enumerate(train_ds), total=EPOCHS):\n with tf.GradientTape() as tape:\n emb, predicted = ae_model(x)\n loss_value = tf.keras.losses.mse(emb, predicted) + sum(model.losses)\n loss_values.append(loss_value)\n gradients = tape.gradient(loss_value, ae_model.trainable_weights)\n optimizer.apply_gradients(zip(gradients, ae_model.trainable_weights))\n if i == EPOCHS:\n break","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-07T13:15:48.498411Z","iopub.execute_input":"2021-07-07T13:15:48.498876Z","iopub.status.idle":"2021-07-07T13:18:19.344253Z","shell.execute_reply.started":"2021-07-07T13:15:48.498829Z","shell.execute_reply":"2021-07-07T13:18:19.343118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 1, figsize=(16,9), dpi= 80)\nmycolors = ['tab:red', 'tab:blue', 'tab:green', 'tab:orange', 'tab:brown', 'tab:grey', 'tab:pink', 'tab:olive'] \n\nloss_values_s = np.std(loss_values, axis=-1)\nloss_values = np.mean(loss_values, axis=-1)\nax.plot(loss_values, '.', alpha=0.7, color=mycolors[1], linewidth=0.5)\nax.fill_between(list(range(EPOCHS + 1)), y1=loss_values + loss_values_s, y2=np.max([np.zeros((EPOCHS+1,)), loss_values - loss_values_s], axis=0), alpha=0.5, color=mycolors[1], linewidth=0)\nax.set_title('AE Loss')\nax.set_xlabel('Iteration')\nax.set_ylabel('Loss');","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:18:19.34692Z","iopub.execute_input":"2021-07-07T13:18:19.347384Z","iopub.status.idle":"2021-07-07T13:18:19.646826Z","shell.execute_reply.started":"2021-07-07T13:18:19.347341Z","shell.execute_reply":"2021-07-07T13:18:19.645692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

2.3. Freezed Pretrained EfficientNetB1 with Encoder
","metadata":{}},{"cell_type":"code","source":"encoder = ae_model_to_e_model(256, emb_model, ae_model)\nencoder.summary()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:18:19.648537Z","iopub.execute_input":"2021-07-07T13:18:19.649181Z","iopub.status.idle":"2021-07-07T13:18:20.890007Z","shell.execute_reply.started":"2021-07-07T13:18:19.64912Z","shell.execute_reply":"2021-07-07T13:18:20.888557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del ae_model, train_ds\ngc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-07T13:18:20.895764Z","iopub.execute_input":"2021-07-07T13:18:20.89634Z","iopub.status.idle":"2021-07-07T13:18:21.162807Z","shell.execute_reply.started":"2021-07-07T13:18:20.89629Z","shell.execute_reply":"2021-07-07T13:18:21.161751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

3. Train dataset analysis
","metadata":{}},{"cell_type":"code","source":"train_ds = get_dataset(files_train_g, batch_size=BATCH_SIZE*2, shuffle = False, repeat = False, labeled=True, return_image_ids=False)\ntest_ds = get_dataset(files_test_g, batch_size=BATCH_SIZE*2, shuffle = False, repeat = False, labeled=False, return_image_ids=True)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:18:21.165912Z","iopub.execute_input":"2021-07-07T13:18:21.166266Z","iopub.status.idle":"2021-07-07T13:18:21.315859Z","shell.execute_reply.started":"2021-07-07T13:18:21.166235Z","shell.execute_reply":"2021-07-07T13:18:21.314753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_emb = np.zeros((EPOCHS_VIS * BATCH_SIZE*2, 2))\ntrain_target = np.zeros((EPOCHS_VIS * BATCH_SIZE*2,))\ntrain_pred = np.zeros((EPOCHS_VIS * BATCH_SIZE*2,))\ntrain_img = np.zeros((BATCH_SIZE * 2, 256, 256, 3))\n\nfor i, (img, target) in tqdm(enumerate(train_ds), total=EPOCHS_VIS - 1):\n train_emb[i*BATCH_SIZE*2: (i + 1) * BATCH_SIZE*2] = encoder.predict(img).astype(np.float16)\n train_pred[i*BATCH_SIZE*2: (i + 1) * BATCH_SIZE*2] = model.predict(img).astype(np.float16)[:, 0]\n train_target[i*BATCH_SIZE*2: (i + 1) * BATCH_SIZE*2] = target.numpy().astype(np.uint8)[:, 0]\n if (i + 1) * BATCH_SIZE*2 <= len(train_img):\n train_img[i*BATCH_SIZE*2: (i + 1) * BATCH_SIZE*2] = img.numpy()\n if i == EPOCHS_VIS - 1:\n break","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-07T13:18:21.319877Z","iopub.execute_input":"2021-07-07T13:18:21.320371Z","iopub.status.idle":"2021-07-07T13:22:49.313339Z","shell.execute_reply.started":"2021-07-07T13:18:21.320324Z","shell.execute_reply":"2021-07-07T13:22:49.311992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(15, 7))\n\nsns.kdeplot(x=train_emb[:, 0], y=train_emb[:, 1], hue=train_target, ax=ax[0])\nax[0].plot(train_emb[:BATCH_SIZE, 0], train_emb[:BATCH_SIZE, 1], '*r')\nax[0].set_xlabel('component 1')\nax[0].set_ylabel('component 2')\nax[0].set_title('Train Set Embeding')\n\npte = train_emb[train_target==1]\nptp = train_pred[train_target==1]\nsns.kdeplot(x=pte[:, 0], y=pte[:, 1], ax=ax[1], alpha=0.4)\ns = ax[1].scatter(x=pte[:, 0], y=pte[:, 1], c=ptp, cmap='inferno')\nax[1].set_xlabel('component 1')\nax[1].set_ylabel('component 2')\nax[1].set_title('Data with gravitation wave probability distribution')\ncbar = plt.colorbar(s)\ncbar.set_label('Probability')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:22:49.315679Z","iopub.execute_input":"2021-07-07T13:22:49.316418Z","iopub.status.idle":"2021-07-07T13:23:31.252434Z","shell.execute_reply.started":"2021-07-07T13:22:49.316352Z","shell.execute_reply":"2021-07-07T13:23:31.250984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, after looking on component 1 & 2 is possible to make a conclusion that on some part of q-transformed spectrograms EfficientNet B1 can't to detect gravitational waves. Let's watch on the three types of q-transformed spectrograms below.","metadata":{}},{"cell_type":"markdown","source":"\n

3.1. Group 1: Without gravitation wave
","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 1, figsize=(7, 7))\n\nimg_groups = {'Without gravitation wave' : []}\n\npte = train_emb[:BATCH_SIZE*2]\nptp = train_pred[:BATCH_SIZE*2]\nptt = train_target[:BATCH_SIZE*2]\n\n_ptp = ptp[ptt == 0]\n_pte = pte[ptt == 0]\n_pti = train_img[ptt==0]\nsns.kdeplot(x=train_emb[:, 0], y=train_emb[:, 1], hue=train_target, ax=ax)\nax.plot(_pte[:, 0], _pte[:, 1], '*r')\nax.set_xlabel('component 1')\nax.set_ylabel('component 2')\nax.set_title('Train Set Embeding')\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-07T13:44:53.272454Z","iopub.execute_input":"2021-07-07T13:44:53.27287Z","iopub.status.idle":"2021-07-07T13:45:19.589143Z","shell.execute_reply.started":"2021-07-07T13:44:53.272833Z","shell.execute_reply":"2021-07-07T13:45:19.587802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(0, min(5, len(_pti))):\n fig, ax = plt.subplots(1, 3, figsize=(21, 7))\n img_groups['Without gravitation wave'].append(_pti[i])\n for j in range(3):\n ax[j].imshow(_pti[i, ..., j])\n fig.suptitle(f'Probability: {_ptp[i]}')\n plt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-07T13:45:19.591181Z","iopub.execute_input":"2021-07-07T13:45:19.591608Z","iopub.status.idle":"2021-07-07T13:45:22.851068Z","shell.execute_reply.started":"2021-07-07T13:45:19.591565Z","shell.execute_reply":"2021-07-07T13:45:22.849739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

3.2. Group 2: With GW & Probability > 0.8
","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 1, figsize=(7, 7))\n\nimg_groups['With GW & Probability > 0.8'] = []\n\npte = train_emb[:BATCH_SIZE*2]\nptp = train_pred[:BATCH_SIZE*2]\nptt = train_target[:BATCH_SIZE*2]\n\nmask = (ptt == 1) & (ptp > 0.8)\n_ptp = ptp[mask]\n_pte = pte[mask]\n_pti = train_img[mask]\nsns.kdeplot(x=train_emb[:, 0], y=train_emb[:, 1], hue=train_target, ax=ax)\nax.plot(_pte[:, 0], _pte[:, 1], '*r')\nax.set_xlabel('component 1')\nax.set_ylabel('component 2')\nax.set_title('Train Set Embeding')\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-07T13:45:22.853747Z","iopub.execute_input":"2021-07-07T13:45:22.854183Z","iopub.status.idle":"2021-07-07T13:45:49.565861Z","shell.execute_reply.started":"2021-07-07T13:45:22.854144Z","shell.execute_reply":"2021-07-07T13:45:49.564765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(0, min(5, len(_pti))):\n fig, ax = plt.subplots(1, 3, figsize=(21, 7))\n img_groups['With GW & Probability > 0.8'].append(_pti[i])\n for j in range(3):\n ax[j].imshow(_pti[i, ..., j])\n fig.suptitle(f'Probability: {_ptp[i]}')\n plt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-07T13:45:49.568069Z","iopub.execute_input":"2021-07-07T13:45:49.568526Z","iopub.status.idle":"2021-07-07T13:45:52.945316Z","shell.execute_reply.started":"2021-07-07T13:45:49.568478Z","shell.execute_reply":"2021-07-07T13:45:52.943984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

3.3. With GW & Probability < 0.8
","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 1, figsize=(7, 7))\n\nimg_groups['With GW & Probability <= 0.8'] = []\n\npte = train_emb[:BATCH_SIZE*2]\nptp = train_pred[:BATCH_SIZE*2]\nptt = train_target[:BATCH_SIZE*2]\n\nmask = (ptt == 1) & (ptp <= 0.8)\n_ptp = ptp[mask]\n_pte = pte[mask]\n_pti = train_img[mask]\nsns.kdeplot(x=train_emb[:, 0], y=train_emb[:, 1], hue=train_target, ax=ax)\nax.plot(_pte[:, 0], _pte[:, 1], '*r')\nax.set_xlabel('component 1')\nax.set_ylabel('component 2')\nax.set_title('Train Set Embeding')\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-07T13:45:52.947206Z","iopub.execute_input":"2021-07-07T13:45:52.947621Z","iopub.status.idle":"2021-07-07T13:46:20.770615Z","shell.execute_reply.started":"2021-07-07T13:45:52.947572Z","shell.execute_reply":"2021-07-07T13:46:20.76948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(0, min(5, len(_pti))):\n fig, ax = plt.subplots(1, 3, figsize=(21, 7))\n img_groups['With GW & Probability <= 0.8'].append(_pti[i])\n for j in range(3):\n ax[j].imshow(_pti[i, ..., j])\n fig.suptitle(f'Probability: {_ptp[i]}')\n plt.show();","metadata":{"execution":{"iopub.status.busy":"2021-07-07T13:46:20.772161Z","iopub.execute_input":"2021-07-07T13:46:20.772538Z","iopub.status.idle":"2021-07-07T13:46:24.687257Z","shell.execute_reply.started":"2021-07-07T13:46:20.772507Z","shell.execute_reply":"2021-07-07T13:46:24.685982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On q-transformed spectrograms, with high predicted probability, gravitational wave patterns are clearly visible. In turn, the spectrograms, for which the model gives a low probability that it contains a gravitational wave signal, are very similar to the spectrograms on which there are no gravitational waves. \nLet's calculate density for each set:","metadata":{}},{"cell_type":"code","source":"pte = train_emb\nptp = train_pred\nptt = train_target\n\nprint('Group 1: Without gravitation wave\\t', sum((ptt == 0)) / len(ptt))\nprint('Group 2: With GW & Probability > 0.8\\t', sum((ptt == 1) & (ptp > 0.8)) / len(ptt))\nprint('Group 3: With GW & Probability <= 0.8\\t', sum((ptt == 1) & (ptp <= 0.8)) / len(ptt))","metadata":{"execution":{"iopub.status.busy":"2021-07-07T14:28:04.491707Z","iopub.execute_input":"2021-07-07T14:28:04.492203Z","iopub.status.idle":"2021-07-07T14:28:04.88935Z","shell.execute_reply.started":"2021-07-07T14:28:04.492171Z","shell.execute_reply":"2021-07-07T14:28:04.88821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

4. Train/Test dataset comparission
","metadata":{}},{"cell_type":"code","source":"test_emb = np.zeros((EPOCHS_VIS * BATCH_SIZE*2, 2))\ntest_pred = np.zeros((EPOCHS_VIS * BATCH_SIZE*2,))\nfor i, (img, idx) in tqdm(enumerate(test_ds), total=EPOCHS_VIS - 1):\n test_emb[i*BATCH_SIZE*2: (i + 1) * BATCH_SIZE*2] = encoder.predict(img).astype(np.float16)\n test_pred[i*BATCH_SIZE*2: (i + 1) * BATCH_SIZE*2] = model.predict(img).astype(np.float16)[:, 0]\n if i == EPOCHS_VIS - 1:\n break","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:46:24.688968Z","iopub.execute_input":"2021-07-07T13:46:24.689373Z","iopub.status.idle":"2021-07-07T13:50:02.600241Z","shell.execute_reply.started":"2021-07-07T13:46:24.689332Z","shell.execute_reply":"2021-07-07T13:50:02.599001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 1, figsize=(8, 7))\nsns.kdeplot(x=test_emb[:, 0], y=test_emb[:, 1], ax=ax, label='Test')\nsns.kdeplot(x=train_emb[:, 0], y=train_emb[:, 1], ax=ax, label='Train')\nax.set_xlabel('component 1')\nax.set_ylabel('component 2')\nfig.suptitle('Train/Test Set Embeding Comparision')\nplt.legend();\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:50:02.603452Z","iopub.execute_input":"2021-07-07T13:50:02.603971Z","iopub.status.idle":"2021-07-07T13:50:56.795024Z","shell.execute_reply.started":"2021-07-07T13:50:02.603909Z","shell.execute_reply":"2021-07-07T13:50:56.793948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"te_hist, te_bins = np.histogram(test_emb[:, 0], bins=96)\ntr_hist, tr_bins = np.histogram(train_emb[:, 0], bins=96, range=[np.min(te_bins), np.max(te_bins)])\n\ntr_hist = tr_hist / np.sum(tr_hist)\nte_hist = te_hist / np.sum(te_hist)\n\nfig, ax = plt.subplots(1, 2, figsize=(15, 7))\nax[0].plot(tr_bins[1:], tr_hist / np.sum(tr_hist), '-')\nax[0].plot(te_bins[1:], te_hist / np.sum(te_hist), '-')\nax[0].set_xlabel('component 1')\nax[0].set_ylabel('densety')\n\np1 = sum(abs(te_hist - tr_hist))\n\nte_hist, te_bins = np.histogram(test_emb[:, 1], bins=96)\ntr_hist, tr_bins = np.histogram(train_emb[:, 1], bins=96, range=[np.min(te_bins), np.max(te_bins)])\n\ntr_hist = tr_hist / np.sum(tr_hist)\nte_hist = te_hist / np.sum(te_hist)\n\nax[1].plot(tr_bins[1:], tr_hist / np.sum(tr_hist), '-')\nax[1].plot(te_bins[1:], te_hist / np.sum(te_hist), '-')\nax[1].set_xlabel('component 2')\nax[1].set_ylabel('densety')\n\np2 = sum(abs(te_hist - tr_hist))\np = 1 - (p1 + p2) / 2\nfig.suptitle(f'Train/Test Set Similarity: {p}');","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:50:56.796893Z","iopub.execute_input":"2021-07-07T13:50:56.79735Z","iopub.status.idle":"2021-07-07T13:50:57.142771Z","shell.execute_reply.started":"2021-07-07T13:50:56.797304Z","shell.execute_reply":"2021-07-07T13:50:57.141352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, base on the graph above is possible to say that train and set is very similar and decisions with the right cross-validation will have a close LB & CV score.","metadata":{}},{"cell_type":"markdown","source":"\n

5.Grad-CAM
","metadata":{}},{"cell_type":"code","source":"model = build_model(256)\nmodel.load_weights('../input/cqt-g2net-efficientnetb1-tpu-training/fold-0.h5')\nmodel_gradcam = build_model_gradcam(256)\nmodel_gradcam.set_weights(model.get_weights())","metadata":{"execution":{"iopub.status.busy":"2021-07-07T13:32:10.146419Z","iopub.execute_input":"2021-07-07T13:32:10.146825Z","iopub.status.idle":"2021-07-07T13:32:19.060126Z","shell.execute_reply.started":"2021-07-07T13:32:10.146793Z","shell.execute_reply":"2021-07-07T13:32:19.059031Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow import keras \n\n# https://keras.io/examples/vision/grad_cam/\ndef make_gradcam_heatmap(img_array, model, last_conv_layer_name, pred_index=None):\n # First, we create a model that maps the input image to the activations\n # of the last conv layer as well as the output predictions\n grad_model = tf.keras.models.Model(\n [model.inputs], [model.get_layer(last_conv_layer_name).output, model.output]\n )\n\n # Then, we compute the gradient of the top predicted class for our input image\n # with respect to the activations of the last conv layer\n with tf.GradientTape() as tape:\n last_conv_layer_output, preds = grad_model(np.expand_dims(img_array, axis=0))\n if pred_index is None:\n pred_index = tf.argmax(preds[0])\n class_channel = preds[:, pred_index]\n\n # This is the gradient of the output neuron (top predicted or chosen)\n # with regard to the output feature map of the last conv layer\n grads = tape.gradient(class_channel, last_conv_layer_output)\n\n # This is a vector where each entry is the mean intensity of the gradient\n # over a specific feature map channel\n pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))\n\n # We multiply each channel in the feature map array\n # by \"how important this channel is\" with regard to the top predicted class\n # then sum all the channels to obtain the heatmap class activation\n last_conv_layer_output = last_conv_layer_output[0]\n heatmap = last_conv_layer_output @ pooled_grads[..., tf.newaxis]\n heatmap = tf.squeeze(heatmap)\n\n # For visualization purpose, we will also normalize the heatmap between 0 & 1\n heatmap = tf.maximum(heatmap, 0) / tf.math.reduce_max(heatmap)\n return heatmap.numpy()\n\ndef display_gradcam(img, heatmap,alpha, ax):\n # Load the original image\n img = keras.preprocessing.image.img_to_array(img)\n\n # Rescale heatmap to a range 0-255\n heatmap = np.uint8(255 * heatmap)\n\n # Use jet colormap to colorize heatmap\n jet = cm.get_cmap(\"jet\")\n\n # Use RGB values of the colormap\n jet_colors = jet(np.arange(256))[:, :3]\n jet_heatmap = jet_colors[heatmap]\n\n # Create an image with RGB colorized heatmap\n jet_heatmap = keras.preprocessing.image.array_to_img(jet_heatmap)\n jet_heatmap = jet_heatmap.resize((img.shape[1], img.shape[0]))\n jet_heatmap = keras.preprocessing.image.img_to_array(jet_heatmap)\n\n # Superimpose the heatmap on original image\n superimposed_img = jet_heatmap * alpha + img * (1 - alpha)\n superimposed_img = keras.preprocessing.image.array_to_img(superimposed_img)\n\n ax.imshow(superimposed_img)","metadata":{"execution":{"iopub.status.busy":"2021-07-07T13:50:57.144682Z","iopub.execute_input":"2021-07-07T13:50:57.14513Z","iopub.status.idle":"2021-07-07T13:50:57.159977Z","shell.execute_reply.started":"2021-07-07T13:50:57.145072Z","shell.execute_reply":"2021-07-07T13:50:57.158497Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for title in img_groups:\n fig, ax = plt.subplots(1, 5, figsize=(25, 5))\n for i in range(0, min(5, len(_pti))):\n img = img_groups[title][i]\n heatmap = make_gradcam_heatmap(img, model_gradcam, 'top_activation')\n display_gradcam(img, heatmap, 0.001, ax[i])\n fig.suptitle(title);\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-07T13:53:15.944083Z","iopub.execute_input":"2021-07-07T13:53:15.94447Z","iopub.status.idle":"2021-07-07T13:53:23.264328Z","shell.execute_reply.started":"2021-07-07T13:53:15.944439Z","shell.execute_reply":"2021-07-07T13:53:23.262942Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On the image above is clearly that trained EfficientNet B1 is confused on a subset of samples with gravitational waves where patterns of gravitational wave are invisible. Are the third group of samples have gravitational waves? Or extracted q-transform features is invalid for this subset of data.","metadata":{}},{"cell_type":"markdown","source":"\n

6. Conclusion
","metadata":{}},{"cell_type":"markdown","source":"* So is possible to make the conclusion that extracted q-transform features are very good features for the separation of Group 2, but this feature is not enough for the separation of Group 3 and Group 1.\n* Base on research above also possible make conclusion that train and test datasets are very similar and decisions with the right cross-validation will have a close LB & CV score. ","metadata":{}},{"cell_type":"markdown","source":"\n

7. References
","metadata":{}},{"cell_type":"markdown","source":"* [Build the right Autoencoder — Tune and Optimize using PCA principles. Part I](https://towardsdatascience.com/build-the-right-autoencoder-tune-and-optimize-using-pca-principles-part-i-1f01f821999b)\n* [Build the right Autoencoder — Tune and Optimize using PCA principles. Part II](https://towardsdatascience.com/build-the-right-autoencoder-tune-and-optimize-using-pca-principles-part-ii-24b9cca69bd6)\n* [Grad-CAM class activation visualization](https://keras.io/examples/vision/grad_cam/)\n* [CQT G2Net EfficientNetB1[TPU Training]](https://www.kaggle.com/miklgr500/cqt-g2net-efficientnetb1-tpu-training)\n* [CQT G2Net EfficientNetB1[TPU Inference] ](https://www.kaggle.com/miklgr500/cqt-g2net-efficientnetb1-tpu-inference)","metadata":{}},{"cell_type":"markdown","source":"P.S. I will be grateful if you can tell me statistical tests for comparing the distributions of random variables from point 4. ","metadata":{}}]}