{ "id": 123752, "title": "Recall Score For Keras Metrics", "url": "/competitions/bengaliai-cv19/discussion/123752", "author_name": "Shangqiu Li", "post_date": "2019-12-30T04:41:18.876000", "votes": 4, "comment_count": 16, "views": 0, "content": "

Did anyone managed to implement a custom metric of recall score with macro average for keras?

", "messages": [ { "id": 706296, "postDate": "2019-12-30T07:49:10.843Z", "content": "

I did it this way:

\n\n

```\nloss_dict={'root': 'categorical_crossentropy',\n 'vowel': 'categorical_crossentropy',\n 'consonant':'categorical_crossentropy'}\nmetrics_dict={ 'root': [keras.metrics.Recall(name='recall')],\n 'vowel': [keras.metrics.Recall(name='recall')],\n 'consonant':[keras.metrics.Recall(name='recall')]}

\n\n

initial_learningrate=2e-3\nmodel.compile(optimizer=RMSprop(lr=initial_learningrate), loss=loss_dict, loss_weights=[2.0,1.0,1.0], \n metrics=metrics_dict)

\n\n

wr_history=[]\nwr_callback = LambdaCallback(on_epoch_end=lambda batch, logs:wr_history.append(0.5*logs['root_recall']+0.25*logs['consonant_recall']+0.25*logs['vowel_recall']))

\n\n

history=model.fit_generator(train_generator,\n valid_generator,\n epochs=epochs,\n steps_per_epoch=train_m//batch_size+1,\n validation_steps = valid_m//batch_size+1,\n callbacks=[wr_callback])

\n\n

history.history['weighted_recall']=wr_history\n```

", "rawMarkdown": "I did it this way:\n\n\n```\nloss_dict={'root': 'categorical_crossentropy',\n 'vowel': 'categorical_crossentropy',\n 'consonant':'categorical_crossentropy'}\nmetrics_dict={ 'root': [keras.metrics.Recall(name='recall')],\n 'vowel': [keras.metrics.Recall(name='recall')],\n 'consonant':[keras.metrics.Recall(name='recall')]}\n\ninitial_learningrate=2e-3\nmodel.compile(optimizer=RMSprop(lr=initial_learningrate), loss=loss_dict, loss_weights=[2.0,1.0,1.0], \n metrics=metrics_dict)\n\nwr_history=[]\nwr_callback = LambdaCallback(on_epoch_end=lambda batch, logs:wr_history.append(0.5*logs['root_recall']+0.25*logs['consonant_recall']+0.25*logs['vowel_recall']))\n\n\nhistory=model.fit_generator(train_generator,\n valid_generator,\n epochs=epochs,\n steps_per_epoch=train_m//batch_size+1,\n validation_steps = valid_m//batch_size+1,\n callbacks=[wr_callback])\n\nhistory.history['weighted_recall']=wr_history\n```", "votes": 4, "replies": [ { "id": 706388, "postDate": "2019-12-30T10:10:19.817Z", "content": "

Not sure I fully understand here. Are you training all 3 models at the same time to be able to get their global recall at each epoch? What is in your train_generator?

", "rawMarkdown": "Not sure I fully understand here. Are you training all 3 models at the same time to be able to get their global recall at each epoch? What is in your `train_generator`?" }, { "id": 706402, "postDate": "2019-12-30T10:42:12.403Z", "content": "

No, it's one model with 3 outputs . My train_generator looks like that:

\n\n

```\ntrain_datagen = ImageDataGenerator(rotation_range = 10,\n width_shift_range = 0.25,\n height_shift_range = 0.25,\n shear_range = 0.1,\n zoom_range = 0.25,\n horizontal_flip = False)

\n\n

columns=[\"root_class\",\"vowel_class\", \"cons_class\"]\nbatch_size = 512\ntrain_generator_from_df = train_datagen.flow_from_dataframe(\n dataframe=train_df,\n directory=Train_dir,\n x_col=\"filename\",\n y_col=columns,\n target_size=(64, 64),\n batch_size=batch_size,\n class_mode=\"multi_output\",\n colormode=\"greyscale\")

\n\n

def split_into_3_outputs(y_batch):\n y1=tf.keras.utils.to_categorical(y_batch[0],168)\n y2=tf.keras.utils.to_categorical(y_batch[1],11)\n y3=tf.keras.utils.to_categorical(y_batch[2],7)

\n\n
return y1,y2,y3\n
\n\n

def aux_data_gen(generator):\n stats = (0.0692, 0.2051)\n while True:\n batch = next(generator)\n batch_x = np.delete(batch[0],[1,2],3)\n batch_x = (batch_x.astype(np.float32)/255.0 - stats[0])/stats[1]\n yield batch_x, split_into_3_outputs(batch[1])

\n\n

train_generator = aux_data_gen(train_generator_from_df)\n```

\n\n

where train_df is data frame with columns [filename,root_class,vowel_class,consonant_class]

\n\n

I use this dataset https://www.kaggle.com/iafoss/grapheme-imgs-128x128, prepared by @iafoss (see details in his kernel: https://www.kaggle.com/iafoss/image-preprocessing-128x128)

", "rawMarkdown": "No, it's one model with 3 outputs . My train_generator looks like that:\n\n\n```\ntrain_datagen = ImageDataGenerator(rotation_range = 10,\n width_shift_range = 0.25,\n height_shift_range = 0.25,\n shear_range = 0.1,\n zoom_range = 0.25,\n horizontal_flip = False)\n\n\ncolumns=[\"root_class\",\"vowel_class\", \"cons_class\"]\nbatch_size = 512\ntrain_generator_from_df = train_datagen.flow_from_dataframe(\n dataframe=train_df,\n directory=Train_dir,\n x_col=\"filename\",\n y_col=columns,\n target_size=(64, 64),\n batch_size=batch_size,\n class_mode=\"multi_output\",\n colormode=\"greyscale\")\n\ndef split_into_3_outputs(y_batch):\n y1=tf.keras.utils.to_categorical(y_batch[0],168)\n y2=tf.keras.utils.to_categorical(y_batch[1],11)\n y3=tf.keras.utils.to_categorical(y_batch[2],7)\n \n return y1,y2,y3\n\ndef aux_data_gen(generator):\n stats = (0.0692, 0.2051)\n while True:\n batch = next(generator)\n batch_x = np.delete(batch[0],[1,2],3)\n batch_x = (batch_x.astype(np.float32)/255.0 - stats[0])/stats[1]\n yield batch_x, split_into_3_outputs(batch[1])\n\ntrain_generator = aux_data_gen(train_generator_from_df)\n```\n\n\nwhere `train_df` is data frame with columns `[filename,root_class,vowel_class,consonant_class]`\n\nI use this dataset https://www.kaggle.com/iafoss/grapheme-imgs-128x128, prepared by @iafoss (see details in his kernel: https://www.kaggle.com/iafoss/image-preprocessing-128x128)", "votes": 1 }, { "id": 706429, "postDate": "2019-12-30T11:08:48.653Z", "content": "

Same question

", "rawMarkdown": "Same question" }, { "id": 706531, "postDate": "2019-12-30T14:25:04.213Z", "content": "

Interesting, I don't think I had seen that before!\nAnd, what are stats and where do the values come from?

", "rawMarkdown": "Interesting, I don't think I had seen that before!\nAnd, what are `stats` and where do the values come from?" }, { "id": 706541, "postDate": "2019-12-30T14:34:19.237Z", "content": "

from here https://www.kaggle.com/iafoss/image-preprocessing-128x128 see last cell

", "rawMarkdown": "from here https://www.kaggle.com/iafoss/image-preprocessing-128x128 see last cell" }, { "id": 706544, "postDate": "2019-12-30T14:38:23.157Z", "content": "

So it's some sort of normalization that you are trying to do?

\n\n

Does it yield better results?

", "rawMarkdown": "So it's some sort of normalization that you are trying to do?\n\nDoes it yield better results?" }, { "id": 706569, "postDate": "2019-12-30T15:12:35.947Z", "content": "

Scaling (or normalizing) inputs makes gradient descent converge faster.

", "rawMarkdown": "Scaling (or normalizing) inputs makes gradient descent converge faster." }, { "id": 706653, "postDate": "2019-12-30T16:55:45.527Z", "content": "

Are you sure this is macro average recall?

", "rawMarkdown": "Are you sure this is macro average recall?" }, { "id": 707277, "postDate": "2019-12-31T14:40:24.890Z", "content": "

Good question! It turns out a few years ago there were problems with keras.metrics.Recall. Details here @thongonary/how-to-compute-f1-score-for-each-epoch-in-keras-a1acd17715a2\">https://medium.com/@thongonary/how-to-compute-f1-score-for-each-epoch-in-keras-a1acd17715a2

\n\n

It will be necessary to check whether this metric is calculated correctly now.

", "rawMarkdown": "Good question! It turns out a few years ago there were problems with keras.metrics.Recall. Details here https://medium.com/@thongonary/how-to-compute-f1-score-for-each-epoch-in-keras-a1acd17715a2 \n\nIt will be necessary to check whether this metric is calculated correctly now. " }, { "id": 707325, "postDate": "2019-12-31T16:34:08.990Z", "content": "

Thanks! It worked after I updated keras in colab.

", "rawMarkdown": "Thanks! It worked after I updated keras in colab." }, { "id": 709681, "postDate": "2020-01-03T20:14:39.353Z", "content": "

The recall method didn't work for me, but this did:

\n\n

```

\n\n

Callback for recall, precision and f1 score calculation

\n\n

class RecallCallback(Callback): \n def init(self):\n self.recall = []

\n\n
def on_epoch_end(self, epoch, logs):\n    eps = np.finfo(np.float32).eps\n    grapheme_recall = logs[\"grapheme_true_positives\"] / (logs[\"grapheme_possible_positives\"] + eps)\n    consonant_recall = logs[\"consonant_true_positives\"] / (logs[\"consonant_possible_positives\"] + eps)\n    vowel_recall = logs[\"vowel_true_positives\"] / (logs[\"vowel_possible_positives\"] + eps)\n    recall = .5 * grapheme_recall + .25 * consonant_recall + .25 * vowel_recall\n    '''\n    For future f1 calc\n    grapheme_precision = logs[\"grapheme_true_positives\"] / (logs[\"grapheme_predicted_positives\"] + eps)\n    consonant_precision = logs[\"consonant_true_positives\"] / (logs[\"consonant_predicted_positives\"] + eps)\n    vowel_precision = logs[\"vowel_true_positives\"] / (logs[\"vowel_predicted_positives\"] + eps)\n    precision = .5 * grapheme_precision + .25 * consonant_precision + .25 * vowel_precision\n    f1 = 2*precision*recall / (precision+recall+eps)\n    '''\n    print (\"\\nRecall (from log) = \", recall)\n    self.recall.append(recall)\n
\n\n

def true_positives(y_true, y_pred):\n return K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))

\n\n

def possible_positives(y_true, y_pred):\n return K.sum(K.round(K.clip(y_true, 0, 1)))

\n\n

def predicted_positives(y_true, y_pred):\n return K.sum(K.round(K.clip(y_pred, 0, 1)))\n\nThen,\n\n model.compile(\n optimizers.Adam(lr=0.0001), \n metrics=['accuracy', true_positives, possible_positives, predicted_positives],\n loss='sparse_categorical_crossentropy'\n\nFinally,\n

\n\n

Callbacks

\n\n

es = tf.keras.callbacks.EarlyStopping(\n monitor='val_loss', min_delta=0.0001, patience=5, verbose=0, mode='auto')\nmc = tf.keras.callbacks.ModelCheckpoint('model.h5', save_best_only=True)\nrecall = RecallCallback()

\n\n

callbacks = [es,mc,recall]

\n\n

train_history = model.fit_generator(\n train_gen,\n steps_per_epoch=train_steps,\n epochs=15,\n validation_data=valid_gen,\n validation_steps=valid_steps,\n callbacks=callbacks\n)\n```

", "rawMarkdown": "The recall method didn't work for me, but this did:\n\n```\n# Callback for recall, precision and f1 score calculation\n\nclass RecallCallback(Callback): \n def __init__(self):\n self.recall = []\n\n def on_epoch_end(self, epoch, logs):\n eps = np.finfo(np.float32).eps\n grapheme_recall = logs[\"grapheme_true_positives\"] / (logs[\"grapheme_possible_positives\"] + eps)\n consonant_recall = logs[\"consonant_true_positives\"] / (logs[\"consonant_possible_positives\"] + eps)\n vowel_recall = logs[\"vowel_true_positives\"] / (logs[\"vowel_possible_positives\"] + eps)\n recall = .5 * grapheme_recall + .25 * consonant_recall + .25 * vowel_recall\n '''\n For future f1 calc\n grapheme_precision = logs[\"grapheme_true_positives\"] / (logs[\"grapheme_predicted_positives\"] + eps)\n consonant_precision = logs[\"consonant_true_positives\"] / (logs[\"consonant_predicted_positives\"] + eps)\n vowel_precision = logs[\"vowel_true_positives\"] / (logs[\"vowel_predicted_positives\"] + eps)\n precision = .5 * grapheme_precision + .25 * consonant_precision + .25 * vowel_precision\n f1 = 2*precision*recall / (precision+recall+eps)\n '''\n print (\"\\nRecall (from log) = \", recall)\n self.recall.append(recall)\n \ndef true_positives(y_true, y_pred):\n return K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n\ndef possible_positives(y_true, y_pred):\n return K.sum(K.round(K.clip(y_true, 0, 1)))\n\ndef predicted_positives(y_true, y_pred):\n return K.sum(K.round(K.clip(y_pred, 0, 1)))\n```\nThen,\n```\n model.compile(\n optimizers.Adam(lr=0.0001), \n metrics=['accuracy', true_positives, possible_positives, predicted_positives],\n loss='sparse_categorical_crossentropy'\n```\nFinally,\n```\n#Callbacks\nes = tf.keras.callbacks.EarlyStopping(\n monitor='val_loss', min_delta=0.0001, patience=5, verbose=0, mode='auto')\nmc = tf.keras.callbacks.ModelCheckpoint('model.h5', save_best_only=True)\nrecall = RecallCallback()\n\ncallbacks = [es,mc,recall]\n\ntrain_history = model.fit_generator(\n train_gen,\n steps_per_epoch=train_steps,\n epochs=15,\n validation_data=valid_gen,\n validation_steps=valid_steps,\n callbacks=callbacks\n)\n```" }, { "id": 709683, "postDate": "2020-01-03T20:16:34.020Z", "content": "

You need to update to keras 2.3.1.

", "rawMarkdown": "You need to update to keras 2.3.1." } ] }, { "id": 706214, "postDate": "2019-12-30T04:41:18.877Z", "content": "

Did anyone managed to implement a custom metric of recall score with macro average for keras?

", "rawMarkdown": "Did anyone managed to implement a custom metric of recall score with macro average for keras? ", "votes": 4 }, { "id": 707122, "postDate": "2019-12-31T09:24:46.233Z", "content": "

Andrey, nice approach. Can you please share how you implement the test generator for inference ?

", "rawMarkdown": "Andrey, nice approach. Can you please share how you implement the test generator for inference ? ", "replies": [ { "id": 707280, "postDate": "2019-12-31T14:45:08.193Z", "content": "

Thanks! My implement the test generator for inference (this is a bit of redone inference by @iafoss https://www.kaggle.com/iafoss/grapheme-fast-ai-starter-inference):

\n\n

```\ndef test_batch_generator(frame, batch_size=64):

\n\n
num_imgs = len(frame)\nstats = (0.0692, 0.2051)\n\nfor batch_start in range(0, num_imgs,batch_size):   \n\n        cur_batch_size = min(num_imgs,batch_start+batch_size)-batch_start\n\n        idx = np.arange(batch_start,batch_start+cur_batch_size)\n        names_batch = frame.iloc[idx,0].values\n        imgs_batch = 255 - frame.iloc[idx, 1:].values.reshape(-1, HEIGHT, WIDTH,1).astype(np.uint8)\n\n        resized_imgs_batch = np.zeros((cur_batch_size,64,64,1))\n\n        for j in range(cur_batch_size):\n            img = (imgs_batch[j,:,:,0]*(255.0/imgs_batch[j,:,:,0].max())).astype(np.uint8)\n            img = crop_resize(img)\n            img = (img.astype(np.float32)/255.0 - stats[0])/stats[1]\n            img = cv2.resize(img,(64,64)).reshape(64,64,1)\n            resized_imgs_batch[j,] = img\n\n        yield resized_imgs_batch,names_batch\n
\n\n

P_TEST_MODE=True

\n\n

TEST_F = ['/kaggle/input/bengaliaicv19feather/test_image_data_0.feather',\n '/kaggle/input/bengaliaicv19feather/test_image_data_1.feather',\n '/kaggle/input/bengaliaicv19feather/test_image_data_2.feather',\n '/kaggle/input/bengaliaicv19feather/test_image_data_3.feather']

\n\n

TEST_P = ['/kaggle/input/bengaliai-cv19/test_image_data_0.parquet',\n '/kaggle/input/bengaliai-cv19/test_image_data_1.parquet',\n '/kaggle/input/bengaliai-cv19/test_image_data_2.parquet',\n '/kaggle/input/bengaliai-cv19/test_image_data_3.parquet']

\n\n

if P_TEST_MODE==True: TEST = TEST_P\nelse : TEST = TEST_F

\n\n

batch_size=1024\nrow_id,target = [],[]

\n\n

for fname in TEST:\n if P_TEST_MODE == True: frame = pd.read_parquet(fname) \n else: frame = pd.read_feather(fname)

\n\n
test_gen = test_batch_generator(frame,batch_size=batch_size)\n\nfor batch_x,batch_name in tqdm(test_gen):\n        batch_predict = model.predict(batch_x)\n        for idx,name in enumerate(batch_name):\n            row_id += [f'{name}_consonant_diacritic',f'{name}_grapheme_root',f'{name}_vowel_diacritic']\n            target += [ np.argmax(batch_predict[2],axis=1)[idx],\n                        np.argmax(batch_predict[0],axis=1)[idx],\n                        np.argmax(batch_predict[1],axis=1)[idx]]\n\nframe.drop(frame.index.values,inplace=True)\n
\n\n

sub_df = pd.DataFrame({'row_id': row_id, 'target': target})\nsub_df.to_csv('submission.csv', index=False)\nsub_df.head()\n```

", "rawMarkdown": "Thanks! My implement the test generator for inference (this is a bit of redone inference by @iafoss https://www.kaggle.com/iafoss/grapheme-fast-ai-starter-inference):\n\n```\ndef test_batch_generator(frame, batch_size=64): \n \n num_imgs = len(frame)\n stats = (0.0692, 0.2051)\n \n for batch_start in range(0, num_imgs,batch_size): \n \n cur_batch_size = min(num_imgs,batch_start+batch_size)-batch_start\n \n idx = np.arange(batch_start,batch_start+cur_batch_size)\n names_batch = frame.iloc[idx,0].values\n imgs_batch = 255 - frame.iloc[idx, 1:].values.reshape(-1, HEIGHT, WIDTH,1).astype(np.uint8)\n \n resized_imgs_batch = np.zeros((cur_batch_size,64,64,1))\n \n for j in range(cur_batch_size):\n img = (imgs_batch[j,:,:,0]*(255.0/imgs_batch[j,:,:,0].max())).astype(np.uint8)\n img = crop_resize(img)\n img = (img.astype(np.float32)/255.0 - stats[0])/stats[1]\n img = cv2.resize(img,(64,64)).reshape(64,64,1)\n resized_imgs_batch[j,] = img\n \n yield resized_imgs_batch,names_batch\n\nP_TEST_MODE=True\n\nTEST_F = ['/kaggle/input/bengaliaicv19feather/test_image_data_0.feather',\n '/kaggle/input/bengaliaicv19feather/test_image_data_1.feather',\n '/kaggle/input/bengaliaicv19feather/test_image_data_2.feather',\n '/kaggle/input/bengaliaicv19feather/test_image_data_3.feather']\n\nTEST_P = ['/kaggle/input/bengaliai-cv19/test_image_data_0.parquet',\n '/kaggle/input/bengaliai-cv19/test_image_data_1.parquet',\n '/kaggle/input/bengaliai-cv19/test_image_data_2.parquet',\n '/kaggle/input/bengaliai-cv19/test_image_data_3.parquet']\n\nif P_TEST_MODE==True: TEST = TEST_P\nelse : TEST = TEST_F\n\nbatch_size=1024\nrow_id,target = [],[]\n\nfor fname in TEST:\n if P_TEST_MODE == True: frame = pd.read_parquet(fname) \n else: frame = pd.read_feather(fname)\n \n test_gen = test_batch_generator(frame,batch_size=batch_size)\n \n for batch_x,batch_name in tqdm(test_gen):\n batch_predict = model.predict(batch_x)\n for idx,name in enumerate(batch_name):\n row_id += [f'{name}_consonant_diacritic',f'{name}_grapheme_root',f'{name}_vowel_diacritic']\n target += [ np.argmax(batch_predict[2],axis=1)[idx],\n np.argmax(batch_predict[0],axis=1)[idx],\n np.argmax(batch_predict[1],axis=1)[idx]]\n \n frame.drop(frame.index.values,inplace=True)\n \nsub_df = pd.DataFrame({'row_id': row_id, 'target': target})\nsub_df.to_csv('submission.csv', index=False)\nsub_df.head()\n```", "votes": 1 } ] }, { "id": 706401, "postDate": "2019-12-30T10:41:13.063Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 706296, "author_name": "Andrey Zotov", "author_url": "", "post_date": "2019-12-30T07:49:10.843000", "content": "

I did it this way:

\n\n

```\nloss_dict={'root': 'categorical_crossentropy',\n 'vowel': 'categorical_crossentropy',\n 'consonant':'categorical_crossentropy'}\nmetrics_dict={ 'root': [keras.metrics.Recall(name='recall')],\n 'vowel': [keras.metrics.Recall(name='recall')],\n 'consonant':[keras.metrics.Recall(name='recall')]}

\n\n

initial_learningrate=2e-3\nmodel.compile(optimizer=RMSprop(lr=initial_learningrate), loss=loss_dict, loss_weights=[2.0,1.0,1.0], \n metrics=metrics_dict)

\n\n

wr_history=[]\nwr_callback = LambdaCallback(on_epoch_end=lambda batch, logs:wr_history.append(0.5*logs['root_recall']+0.25*logs['consonant_recall']+0.25*logs['vowel_recall']))

\n\n

history=model.fit_generator(train_generator,\n valid_generator,\n epochs=epochs,\n steps_per_epoch=train_m//batch_size+1,\n validation_steps = valid_m//batch_size+1,\n callbacks=[wr_callback])

\n\n

history.history['weighted_recall']=wr_history\n```

", "votes": 4, "replies": [ { "id": 706388, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2019-12-30T10:10:19.817000", "content": "

Not sure I fully understand here. Are you training all 3 models at the same time to be able to get their global recall at each epoch? What is in your train_generator?

", "votes": 0, "replies": [] }, { "id": 706402, "author_name": "Andrey Zotov", "author_url": "", "post_date": "2019-12-30T10:42:12.403000", "content": "

No, it's one model with 3 outputs . My train_generator looks like that:

\n\n

```\ntrain_datagen = ImageDataGenerator(rotation_range = 10,\n width_shift_range = 0.25,\n height_shift_range = 0.25,\n shear_range = 0.1,\n zoom_range = 0.25,\n horizontal_flip = False)

\n\n

columns=[\"root_class\",\"vowel_class\", \"cons_class\"]\nbatch_size = 512\ntrain_generator_from_df = train_datagen.flow_from_dataframe(\n dataframe=train_df,\n directory=Train_dir,\n x_col=\"filename\",\n y_col=columns,\n target_size=(64, 64),\n batch_size=batch_size,\n class_mode=\"multi_output\",\n colormode=\"greyscale\")

\n\n

def split_into_3_outputs(y_batch):\n y1=tf.keras.utils.to_categorical(y_batch[0],168)\n y2=tf.keras.utils.to_categorical(y_batch[1],11)\n y3=tf.keras.utils.to_categorical(y_batch[2],7)

\n\n
return y1,y2,y3\n
\n\n

def aux_data_gen(generator):\n stats = (0.0692, 0.2051)\n while True:\n batch = next(generator)\n batch_x = np.delete(batch[0],[1,2],3)\n batch_x = (batch_x.astype(np.float32)/255.0 - stats[0])/stats[1]\n yield batch_x, split_into_3_outputs(batch[1])

\n\n

train_generator = aux_data_gen(train_generator_from_df)\n```

\n\n

where train_df is data frame with columns [filename,root_class,vowel_class,consonant_class]

\n\n

I use this dataset https://www.kaggle.com/iafoss/grapheme-imgs-128x128, prepared by @iafoss (see details in his kernel: https://www.kaggle.com/iafoss/image-preprocessing-128x128)

", "votes": 1, "replies": [] }, { "id": 706429, "author_name": "", "author_url": "", "post_date": "2019-12-30T11:08:48.653000", "content": "

Same question

", "votes": 0, "replies": [] }, { "id": 706531, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2019-12-30T14:25:04.213000", "content": "

Interesting, I don't think I had seen that before!\nAnd, what are stats and where do the values come from?

", "votes": 0, "replies": [] }, { "id": 706541, "author_name": "Andrey Zotov", "author_url": "", "post_date": "2019-12-30T14:34:19.237000", "content": "

from here https://www.kaggle.com/iafoss/image-preprocessing-128x128 see last cell

", "votes": 0, "replies": [] }, { "id": 706544, "author_name": "Maxime Lenormand", "author_url": "", "post_date": "2019-12-30T14:38:23.157000", "content": "

So it's some sort of normalization that you are trying to do?

\n\n

Does it yield better results?

", "votes": 0, "replies": [] }, { "id": 706569, "author_name": "Andrey Zotov", "author_url": "", "post_date": "2019-12-30T15:12:35.947000", "content": "

Scaling (or normalizing) inputs makes gradient descent converge faster.

", "votes": 0, "replies": [] }, { "id": 706653, "author_name": "Shangqiu Li", "author_url": "", "post_date": "2019-12-30T16:55:45.527000", "content": "

Are you sure this is macro average recall?

", "votes": 0, "replies": [] }, { "id": 707277, "author_name": "Andrey Zotov", "author_url": "", "post_date": "2019-12-31T14:40:24.890000", "content": "

Good question! It turns out a few years ago there were problems with keras.metrics.Recall. Details here @thongonary/how-to-compute-f1-score-for-each-epoch-in-keras-a1acd17715a2\">https://medium.com/@thongonary/how-to-compute-f1-score-for-each-epoch-in-keras-a1acd17715a2

\n\n

It will be necessary to check whether this metric is calculated correctly now.

", "votes": 0, "replies": [] }, { "id": 707325, "author_name": "Shangqiu Li", "author_url": "", "post_date": "2019-12-31T16:34:08.990000", "content": "

Thanks! It worked after I updated keras in colab.

", "votes": 0, "replies": [] }, { "id": 709681, "author_name": "Larry Schuster", "author_url": "", "post_date": "2020-01-03T20:14:39.353000", "content": "

The recall method didn't work for me, but this did:

\n\n

```

\n\n

Callback for recall, precision and f1 score calculation

\n\n

class RecallCallback(Callback): \n def init(self):\n self.recall = []

\n\n
def on_epoch_end(self, epoch, logs):\n    eps = np.finfo(np.float32).eps\n    grapheme_recall = logs[\"grapheme_true_positives\"] / (logs[\"grapheme_possible_positives\"] + eps)\n    consonant_recall = logs[\"consonant_true_positives\"] / (logs[\"consonant_possible_positives\"] + eps)\n    vowel_recall = logs[\"vowel_true_positives\"] / (logs[\"vowel_possible_positives\"] + eps)\n    recall = .5 * grapheme_recall + .25 * consonant_recall + .25 * vowel_recall\n    '''\n    For future f1 calc\n    grapheme_precision = logs[\"grapheme_true_positives\"] / (logs[\"grapheme_predicted_positives\"] + eps)\n    consonant_precision = logs[\"consonant_true_positives\"] / (logs[\"consonant_predicted_positives\"] + eps)\n    vowel_precision = logs[\"vowel_true_positives\"] / (logs[\"vowel_predicted_positives\"] + eps)\n    precision = .5 * grapheme_precision + .25 * consonant_precision + .25 * vowel_precision\n    f1 = 2*precision*recall / (precision+recall+eps)\n    '''\n    print (\"\\nRecall (from log) = \", recall)\n    self.recall.append(recall)\n
\n\n

def true_positives(y_true, y_pred):\n return K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))

\n\n

def possible_positives(y_true, y_pred):\n return K.sum(K.round(K.clip(y_true, 0, 1)))

\n\n

def predicted_positives(y_true, y_pred):\n return K.sum(K.round(K.clip(y_pred, 0, 1)))\n\nThen,\n\n model.compile(\n optimizers.Adam(lr=0.0001), \n metrics=['accuracy', true_positives, possible_positives, predicted_positives],\n loss='sparse_categorical_crossentropy'\n\nFinally,\n

\n\n

Callbacks

\n\n

es = tf.keras.callbacks.EarlyStopping(\n monitor='val_loss', min_delta=0.0001, patience=5, verbose=0, mode='auto')\nmc = tf.keras.callbacks.ModelCheckpoint('model.h5', save_best_only=True)\nrecall = RecallCallback()

\n\n

callbacks = [es,mc,recall]

\n\n

train_history = model.fit_generator(\n train_gen,\n steps_per_epoch=train_steps,\n epochs=15,\n validation_data=valid_gen,\n validation_steps=valid_steps,\n callbacks=callbacks\n)\n```

", "votes": 0, "replies": [] }, { "id": 709683, "author_name": "Shangqiu Li", "author_url": "", "post_date": "2020-01-03T20:16:34.020000", "content": "

You need to update to keras 2.3.1.

", "votes": 0, "replies": [] } ] }, { "id": 707122, "author_name": "Ioannis M", "author_url": "", "post_date": "2019-12-31T09:24:46.233000", "content": "

Andrey, nice approach. Can you please share how you implement the test generator for inference ?

", "votes": 0, "replies": [ { "id": 707280, "author_name": "Andrey Zotov", "author_url": "", "post_date": "2019-12-31T14:45:08.193000", "content": "

Thanks! My implement the test generator for inference (this is a bit of redone inference by @iafoss https://www.kaggle.com/iafoss/grapheme-fast-ai-starter-inference):

\n\n

```\ndef test_batch_generator(frame, batch_size=64):

\n\n
num_imgs = len(frame)\nstats = (0.0692, 0.2051)\n\nfor batch_start in range(0, num_imgs,batch_size):   \n\n        cur_batch_size = min(num_imgs,batch_start+batch_size)-batch_start\n\n        idx = np.arange(batch_start,batch_start+cur_batch_size)\n        names_batch = frame.iloc[idx,0].values\n        imgs_batch = 255 - frame.iloc[idx, 1:].values.reshape(-1, HEIGHT, WIDTH,1).astype(np.uint8)\n\n        resized_imgs_batch = np.zeros((cur_batch_size,64,64,1))\n\n        for j in range(cur_batch_size):\n            img = (imgs_batch[j,:,:,0]*(255.0/imgs_batch[j,:,:,0].max())).astype(np.uint8)\n            img = crop_resize(img)\n            img = (img.astype(np.float32)/255.0 - stats[0])/stats[1]\n            img = cv2.resize(img,(64,64)).reshape(64,64,1)\n            resized_imgs_batch[j,] = img\n\n        yield resized_imgs_batch,names_batch\n
\n\n

P_TEST_MODE=True

\n\n

TEST_F = ['/kaggle/input/bengaliaicv19feather/test_image_data_0.feather',\n '/kaggle/input/bengaliaicv19feather/test_image_data_1.feather',\n '/kaggle/input/bengaliaicv19feather/test_image_data_2.feather',\n '/kaggle/input/bengaliaicv19feather/test_image_data_3.feather']

\n\n

TEST_P = ['/kaggle/input/bengaliai-cv19/test_image_data_0.parquet',\n '/kaggle/input/bengaliai-cv19/test_image_data_1.parquet',\n '/kaggle/input/bengaliai-cv19/test_image_data_2.parquet',\n '/kaggle/input/bengaliai-cv19/test_image_data_3.parquet']

\n\n

if P_TEST_MODE==True: TEST = TEST_P\nelse : TEST = TEST_F

\n\n

batch_size=1024\nrow_id,target = [],[]

\n\n

for fname in TEST:\n if P_TEST_MODE == True: frame = pd.read_parquet(fname) \n else: frame = pd.read_feather(fname)

\n\n
test_gen = test_batch_generator(frame,batch_size=batch_size)\n\nfor batch_x,batch_name in tqdm(test_gen):\n        batch_predict = model.predict(batch_x)\n        for idx,name in enumerate(batch_name):\n            row_id += [f'{name}_consonant_diacritic',f'{name}_grapheme_root',f'{name}_vowel_diacritic']\n            target += [ np.argmax(batch_predict[2],axis=1)[idx],\n                        np.argmax(batch_predict[0],axis=1)[idx],\n                        np.argmax(batch_predict[1],axis=1)[idx]]\n\nframe.drop(frame.index.values,inplace=True)\n
\n\n

sub_df = pd.DataFrame({'row_id': row_id, 'target': target})\nsub_df.to_csv('submission.csv', index=False)\nsub_df.head()\n```

", "votes": 1, "replies": [] } ] }, { "id": 706401, "author_name": "", "author_url": "", "post_date": "2019-12-30T10:41:13.063000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "706296": "I did it this way:\n\n\n```\nloss_dict={'root': 'categorical_crossentropy',\n 'vowel': 'categorical_crossentropy',\n 'consonant':'categorical_crossentropy'}\nmetrics_dict={ 'root': [keras.metrics.Recall(name='recall')],\n 'vowel': [keras.metrics.Recall(name='recall')],\n 'consonant':[keras.metrics.Recall(name='recall')]}\n\ninitial_learningrate=2e-3\nmodel.compile(optimizer=RMSprop(lr=initial_learningrate), loss=loss_dict, loss_weights=[2.0,1.0,1.0], \n metrics=metrics_dict)\n\nwr_history=[]\nwr_callback = LambdaCallback(on_epoch_end=lambda batch, logs:wr_history.append(0.5*logs['root_recall']+0.25*logs['consonant_recall']+0.25*logs['vowel_recall']))\n\n\nhistory=model.fit_generator(train_generator,\n valid_generator,\n epochs=epochs,\n steps_per_epoch=train_m//batch_size+1,\n validation_steps = valid_m//batch_size+1,\n callbacks=[wr_callback])\n\nhistory.history['weighted_recall']=wr_history\n```", "706214": "Did anyone managed to implement a custom metric of recall score with macro average for keras? ", "707122": "Andrey, nice approach. Can you please share how you implement the test generator for inference ? ", "706401": "" } }