Please see the post by @hengck23 for understanding Ugly Duckling Concept
\n\nSince outliers among images of a given patient are more likely to be Malignant Melanomas, but similar looking lesions in another patient may not be malignant, we can improve accuracy of predictions by considering the context (i.e. patient_id).
\n\nHowever, the numbers per patient are very small. Please see below contextual analysis:\n 2056 unique patient_id in train data 33126 images\n 428 patients have malignant melanoma, out of which\n - 319 have only 1 malignant lesion and no benign images\n - 109 have both types (benign and malignant)\n These 109 patient images with both types:\n - have 5 to 78 total images (min, max) per patient\n - where 2 to 8 are malignant AND\n - another 3 to 76 are benign images per patient
\n\nFor such small groups of data, we can use Dixon's Q test (see wiki here), where\n Q = abs(gap) / range\ni.e. after obtaining sorted model predictions, we can group them by patient_id, look for gaps (in the sorted list) and identify outliers.
Please see the post by @hengck23 for understanding Ugly Duckling Concept
\n\nSince outliers among images of a given patient are more likely to be Malignant Melanomas, but similar looking lesions in another patient may not be malignant, we can improve accuracy of predictions by considering the context (i.e. patient_id).
\n\nHowever, the numbers per patient are very small. Please see below contextual analysis:\n 2056 unique patient_id in train data 33126 images\n 428 patients have malignant melanoma, out of which\n - 319 have only 1 malignant lesion and no benign images\n - 109 have both types (benign and malignant)\n These 109 patient images with both types:\n - have 5 to 78 total images (min, max) per patient\n - where 2 to 8 are malignant AND\n - another 3 to 76 are benign images per patient
\n\nFor such small groups of data, we can use Dixon's Q test (see wiki here), where\n Q = abs(gap) / range\ni.e. after obtaining sorted model predictions, we can group them by patient_id, look for gaps (in the sorted list) and identify outliers.
Thank you for the pointers, your observations definitely lead us in the right direction.
\n\nAs a sort of sanity check, however, I wanted to point out that I am getting different numbers than you: out of those 2056 patients, I get 428 patients with positive (malignant melanoma) samples, but 427 with both positive and negative samples. In other words, I find there is only one patient with just positive sample(s).
\n\nI was wondering if you wouldn't mind sharing your logic. Here is mine:
\n\n\npatient_ids = sorted(data.patient_id.unique())\ndata_pat_tar_gb = data.groupby(['patient_id', 'target'])\ndata_pat_tar_dict = data_pat_tar_gb.groups\npatients_pos = [p for p in patient_ids if (p, 1) in data_pat_tar_dict]\npatients_neg = [p for p in patient_ids if (p, 0) in data_pat_tar_dict]\npatients_both = set(patients_pos).intersection(patients_neg)\n
Thank you for the pointers, your observations definitely lead us in the right direction.
\n\nAs a sort of sanity check, however, I wanted to point out that I am getting different numbers than you: out of those 2056 patients, I get 428 patients with positive (malignant melanoma) samples, but 427 with both positive and negative samples. In other words, I find there is only one patient with just positive sample(s).
\n\nI was wondering if you wouldn't mind sharing your logic. Here is mine:
\n\n\npatient_ids = sorted(data.patient_id.unique())\ndata_pat_tar_gb = data.groupby(['patient_id', 'target'])\ndata_pat_tar_dict = data_pat_tar_gb.groups\npatients_pos = [p for p in patient_ids if (p, 1) in data_pat_tar_dict]\npatients_neg = [p for p in patient_ids if (p, 0) in data_pat_tar_dict]\npatients_both = set(patients_pos).intersection(patients_neg)\n