{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from fastai.vision.all import *","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-06-05T06:11:58.688798Z","iopub.execute_input":"2021-06-05T06:11:58.689196Z","iopub.status.idle":"2021-06-05T06:11:58.695083Z","shell.execute_reply.started":"2021-06-05T06:11:58.689166Z","shell.execute_reply":"2021-06-05T06:11:58.693753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### LB Probes\n\n#### Probe 1: study (1,1,2,2 invalid boxes) and image (1,1,2,2 invalid boxes):  \n\nLB:0.0 Expectation:0.0\n\n#### Probe 2: study (1,1,2,2 invalid boxes) and image (none,1,0,0,1,1): \n\nLB:0.0 Expectation:1/6 (Assuming equal class weights mAP)\n\nLB: 0.0 also supports the existing problem with image level score calculations. Many say it only contributes 0.001 but infact its not contributing at all, not sure when we will have an explanation about this.\n\n#### Probe 3: study (negative,1,1,0,0) and image (1,1,2,2 boxes):  \n\nLB:0.050 Expectation:1/6 (Assuming equal class weights mAP)\n\nThis means mAP is weighted by the number of samples of that class in test set. Probe 4 and 5 supports this idea.\n\n#### Probe 4: study (typical,1,1,0,0) and image (1,1,2,2 boxes):  \n\nLB:0.079 Expectation:1/6 (Assuming equal class weights mAP)\n\nThis means mAP is weighted.\n\nLB negative probe / LB typical probe = 0.050 / 0.079 = 0.63\n\nno. of negative in train / no. of typical in train =  1676 / 2855 = 0.58\n\nWe can see that ratios of LB scores and number of samples are very close. They are not exactly same since class distributions of training and test are not same, we probably have more negatives in test set.\n\n#### Probe 4: study (indeterminate,1,1,0,0) and image (1,1,2,2 boxes):  \n\nLB:0.031 Expectation:1/6 (Assuming equal class weights mAP)\n\nThis means mAP is weighted.\n\nLB negative probe / LB indeterminate probe = 0.050 / 0.031 = 1.61\n\nno. of negative in train / no. of indeterminate in train =  1676 / 2855 = 1.59\n\n#### Probe 5: study (atypical,1,1,0,0) and image (1,1,2,2 boxes):  \n\nLB:0.031 Expectation:1/6 (Assuming equal class weights mAP)\n\nThis means mAP is weighted.\n\nLB negative probe / LB atypical probe = 0.050 / X = Y (Tomorrow: out of submissions)\n\nno. of negative in train / no. of atypical in train =  1676 / 2855 = 3.53\n\nX should be close to 0.050 / 3.53 ~= 0.14\n\n\n\n### What are weights for image and study level predictions?\n\nAbove we see that mAP is not weigthed equally, hence not 1/6. We can see the relative contributions within study level classes, but image level classes might be differently weighted compared to study level classes.\n\n\nWe kind of now know that study level predictions are weighted proportionally to their number of samples, if we also assume this for all the 6 classes then here is a rough calculation:\n\n\nLB (probe 3): 0.050 -  Expectation: no. of negative in train / all train samples = 0.136\n\nLB (probe 4): 0.079 -  Expectation: no. of negative in train / all train samples = 0.230\n\nLB (probe 4): 0.031 -  Expectation: no. of negative in train / all train samples = 0.084\n\n\nWhen we assume equal weight between study and image level predictions numbers don't match. Here if we assume a weight of 0.35 for study level predictions:\n\nLB (probe 3): 0.050 -  Expectation: 0.35*(no. of negative in train / all train samples) = 0.047\n\nLB (probe 4): 0.079 -  Expectation: 0.35*(no. of negative in train / all train samples) = 0.080\n\nLB (probe 4): 0.031 -  Expectation: 0.35*(no. of negative in train / all train samples) = 0.029\n\nAgain numbers are slightly different because train and test distribution differences.\n\n\n### Evaluation Formula Hypothesis\n\nScore: 0.35x(Weighted mAP Study) + 0.65x(Weighted mAP Image) ???\n\n\n\nP.S. We can't do much probing around images since evaluation is currently broken.","metadata":{}},{"cell_type":"code","source":"len(train_image_df)/len(image_)","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:35:28.740592Z","iopub.execute_input":"2021-06-05T06:35:28.742846Z","iopub.status.idle":"2021-06-05T06:35:28.748343Z","shell.execute_reply.started":"2021-06-05T06:35:28.742801Z","shell.execute_reply":"2021-06-05T06:35:28.747126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image_df = pd.read_csv(\"../input/siim-covid19-detection/train_image_level.csv\")\ntrain_study_df = pd.read_csv(\"../input/siim-covid19-detection/train_study_level.csv\")","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:11:59.095787Z","iopub.execute_input":"2021-06-05T06:11:59.096205Z","iopub.status.idle":"2021-06-05T06:11:59.143106Z","shell.execute_reply.started":"2021-06-05T06:11:59.096166Z","shell.execute_reply":"2021-06-05T06:11:59.141865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_image_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:11:59.352013Z","iopub.execute_input":"2021-06-05T06:11:59.352410Z","iopub.status.idle":"2021-06-05T06:11:59.366446Z","shell.execute_reply.started":"2021-06-05T06:11:59.352378Z","shell.execute_reply":"2021-06-05T06:11:59.365500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_study_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:11:59.463176Z","iopub.execute_input":"2021-06-05T06:11:59.463689Z","iopub.status.idle":"2021-06-05T06:11:59.475676Z","shell.execute_reply.started":"2021-06-05T06:11:59.463657Z","shell.execute_reply":"2021-06-05T06:11:59.474425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub_df = pd.read_csv(\"../input/siim-covid19-detection/sample_submission.csv\")\nstudy_sub_df = sample_sub_df[sample_sub_df['id'].str.contains(\"study\")]\nimage_sub_df = sample_sub_df[sample_sub_df['id'].str.contains(\"image\")]","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:11:59.796268Z","iopub.execute_input":"2021-06-05T06:11:59.796844Z","iopub.status.idle":"2021-06-05T06:11:59.815498Z","shell.execute_reply.started":"2021-06-05T06:11:59.796809Z","shell.execute_reply":"2021-06-05T06:11:59.814122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"label_counts = Counter(np.where(array(train_study_df.iloc[:,1:]))[1]); label_counts","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:12:00.228685Z","iopub.execute_input":"2021-06-05T06:12:00.229120Z","iopub.status.idle":"2021-06-05T06:12:00.240299Z","shell.execute_reply.started":"2021-06-05T06:12:00.229086Z","shell.execute_reply":"2021-06-05T06:12:00.239422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(label_counts[2]/(len(train_image_df)+len(train_study_df)))*0.35","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:43:41.613606Z","iopub.execute_input":"2021-06-05T06:43:41.614054Z","iopub.status.idle":"2021-06-05T06:43:41.621564Z","shell.execute_reply.started":"2021-06-05T06:43:41.614009Z","shell.execute_reply":"2021-06-05T06:43:41.620358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"label_counts[0]/label_counts[1], 0.05/0.079","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:26:35.047818Z","iopub.execute_input":"2021-06-05T06:26:35.048396Z","iopub.status.idle":"2021-06-05T06:26:35.054732Z","shell.execute_reply.started":"2021-06-05T06:26:35.048362Z","shell.execute_reply":"2021-06-05T06:26:35.053951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"label_counts[0]/label_counts[3], 0.05/0.031","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:31:11.163044Z","iopub.execute_input":"2021-06-05T06:31:11.163450Z","iopub.status.idle":"2021-06-05T06:31:11.171119Z","shell.execute_reply.started":"2021-06-05T06:31:11.163417Z","shell.execute_reply":"2021-06-05T06:31:11.169885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def invalid_label_func(o): return \" \".join(o.split()[:-4] + ['1','1','2','2'])\ndef study_label_func(o,l): return f\"{l} 1 0 0 1 1\"","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:12:13.792694Z","iopub.execute_input":"2021-06-05T06:12:13.793263Z","iopub.status.idle":"2021-06-05T06:12:13.799647Z","shell.execute_reply.started":"2021-06-05T06:12:13.793231Z","shell.execute_reply":"2021-06-05T06:12:13.798405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"study_sub_df.loc[:,'PredictionString'] = study_sub_df['PredictionString'].apply(partial(study_label_func,l='atypical')).values\nimage_sub_df.loc[:,'PredictionString'] = image_sub_df['PredictionString'].apply(invalid_label_func).values","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:12:14.175628Z","iopub.execute_input":"2021-06-05T06:12:14.176182Z","iopub.status.idle":"2021-06-05T06:12:14.187389Z","shell.execute_reply.started":"2021-06-05T06:12:14.176147Z","shell.execute_reply":"2021-06-05T06:12:14.186370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub_df = pd.concat([study_sub_df, image_sub_df])","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:12:16.018789Z","iopub.execute_input":"2021-06-05T06:12:16.019147Z","iopub.status.idle":"2021-06-05T06:12:16.025594Z","shell.execute_reply.started":"2021-06-05T06:12:16.019117Z","shell.execute_reply":"2021-06-05T06:12:16.024520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub_df.to_csv(\"submission.csv\",index=False)","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:12:21.530595Z","iopub.execute_input":"2021-06-05T06:12:21.531176Z","iopub.status.idle":"2021-06-05T06:12:21.544977Z","shell.execute_reply.started":"2021-06-05T06:12:21.531131Z","shell.execute_reply":"2021-06-05T06:12:21.544036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub_df","metadata":{"execution":{"iopub.status.busy":"2021-06-05T06:12:22.034516Z","iopub.execute_input":"2021-06-05T06:12:22.035170Z","iopub.status.idle":"2021-06-05T06:12:22.050838Z","shell.execute_reply.started":"2021-06-05T06:12:22.035116Z","shell.execute_reply":"2021-06-05T06:12:22.049102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}