![\"example\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/379292/10248/example.png\")
Congrats to all the winners! and I'd like to thank Google and Kaggle for this interesting competition.
\n\nThis is My brief summary.
\n\nI used yolo for object detection with following modification.
\n\nRelation data(challenge-2018-train-vrd.csv) is mapped to all combinations of human labeled BBs(challenge-2018-train-vrd-bbox.csv) and the data is used as training data of model2.
\n\n
I made training data by mapping the ground truth BBOX(challenge-2018-train-vrd-bbox.csv) to ground truth relational data(challenge-2018-train-vrd.csv), giving a target.
\n\nExample:
\n\nbbox data:
\n\nimage1,bbox1,Man\n image1,bbox2,Guiter\n image1,bbox3,Chair\n\n\n
Relationship data:
\n\nimage1, bbox2, bbox2, Guitar, Wooden, is\n image1, bbox3, bbox3, Chair, Wooden, is\n image1, bbox1, bbox2, Man, Guitar, hold\n\n\n
then, training data would be
\n\nimage1,bbox1,bbox1, Man,None,is \n image1,bbox2,bbox2, Guiter,Wooden,is\n image1,bbox3,bbox3, Chair ,Wooden,is \n\n\n
here, target is None if the data is not in Relationship data.
\n\nFirst, I created all pairs of BBOX in the same image with some filter.\nThen, I made training data by mapping the pairs to ground truth relational data, giving a target.
\n\nExample:\nbbox data :
\n\nimage1,bbox1,Man\n image1,bbox2,Guiter\n image1,bbox3,Chair\n\n\n
Relationship data:
\n\nimage1,bbox1,bbox2,Man,Guiter,hold\n\n\n
Then training data would be
\n\nimage1,bbox1,bbox2,Man,Guiter,hold\n image1,bbox1,bbox3,Man,Chair,None\n image1,bbox2,bbox1,Guiter,Man,None\n image1,bbox2,bbox3,Guiter,Chair,None\n image1,bbox3,bbox1,Chair,Man,None\n image1,bbox3,bbox2,Chair,Guiter,None\n\n\n
here, target is None if the data is not in Relationship data.
\n\nRelation data(challenge-2018-train-vrd.csv) is mapped to all combinations of predicted BBs(output of model1), and the data is used as training data of model3.
\n\nModel2 and model3 can be merged to one NN model.
\n\nThe data set of this competition has the following three characteristics.
\n\nFor example, there are cases where cats are not checked even if there is a cat in the image.\nThis causes false penalty if the model detects a cat during learning.
\n\nFor example, cat class is a child class of the Animal class, so a BBOX of cat is also a BBOX of Animal.
\n\nTo handle these better, I first deleted the confidence term from YOLO's objective function.\nBy deleting this term, class probability will be responsible for confidence.\nOnce class probability also have confidence role, this make it possible to mask unchecked classes.\nBy masking the cat class in the example above it becomes possible to eliminate the penalty when cat is detected by the model.
\n\nHowever, It is unlikely that different classes of boxes will be in the same area(same place, and same size, and same aspect).\nSo, I did not mask for the same area where some bbox exists.
\n\nBut, there is an exception to this, as bbox of child class can exist in the same area(Animal can be a cat as well).\nSo I masked the child class of the corresponding BBOX.
\n\nIn order to cope with an imbalance of (3), I removed same images.\nBut sampling images is not enough to balance the classes, so I introduced class weight.
", "messages": [ { "id": "379292", "postDate": "08/31/2018 05:59:41", "content": "Congrats to all the winners! and I'd like to thank Google and Kaggle for this interesting competition.
\n\nThis is My brief summary.
\n\nI used yolo for object detection with following modification.
\n\nRelation data(challenge-2018-train-vrd.csv) is mapped to all combinations of human labeled BBs(challenge-2018-train-vrd-bbox.csv) and the data is used as training data of model2.
\n\n
I made training data by mapping the ground truth BBOX(challenge-2018-train-vrd-bbox.csv) to ground truth relational data(challenge-2018-train-vrd.csv), giving a target.
\n\nExample:
\n\nbbox data:
\n\nimage1,bbox1,Man\n image1,bbox2,Guiter\n image1,bbox3,Chair\n\n\n
Relationship data:
\n\nimage1, bbox2, bbox2, Guitar, Wooden, is\n image1, bbox3, bbox3, Chair, Wooden, is\n image1, bbox1, bbox2, Man, Guitar, hold\n\n\n
then, training data would be
\n\nimage1,bbox1,bbox1, Man,None,is \n image1,bbox2,bbox2, Guiter,Wooden,is\n image1,bbox3,bbox3, Chair ,Wooden,is \n\n\n
here, target is None if the data is not in Relationship data.
\n\nFirst, I created all pairs of BBOX in the same image with some filter.\nThen, I made training data by mapping the pairs to ground truth relational data, giving a target.
\n\nExample:\nbbox data :
\n\nimage1,bbox1,Man\n image1,bbox2,Guiter\n image1,bbox3,Chair\n\n\n
Relationship data:
\n\nimage1,bbox1,bbox2,Man,Guiter,hold\n\n\n
Then training data would be
\n\nimage1,bbox1,bbox2,Man,Guiter,hold\n image1,bbox1,bbox3,Man,Chair,None\n image1,bbox2,bbox1,Guiter,Man,None\n image1,bbox2,bbox3,Guiter,Chair,None\n image1,bbox3,bbox1,Chair,Man,None\n image1,bbox3,bbox2,Chair,Guiter,None\n\n\n
here, target is None if the data is not in Relationship data.
\n\nRelation data(challenge-2018-train-vrd.csv) is mapped to all combinations of predicted BBs(output of model1), and the data is used as training data of model3.
\n\nModel2 and model3 can be merged to one NN model.
\n\nThe data set of this competition has the following three characteristics.
\n\nFor example, there are cases where cats are not checked even if there is a cat in the image.\nThis causes false penalty if the model detects a cat during learning.
\n\nFor example, cat class is a child class of the Animal class, so a BBOX of cat is also a BBOX of Animal.
\n\nTo handle these better, I first deleted the confidence term from YOLO's objective function.\nBy deleting this term, class probability will be responsible for confidence.\nOnce class probability also have confidence role, this make it possible to mask unchecked classes.\nBy masking the cat class in the example above it becomes possible to eliminate the penalty when cat is detected by the model.
\n\nHowever, It is unlikely that different classes of boxes will be in the same area(same place, and same size, and same aspect).\nSo, I did not mask for the same area where some bbox exists.
\n\nBut, there is an exception to this, as bbox of child class can exist in the same area(Animal can be a cat as well).\nSo I masked the child class of the corresponding BBOX.
\n\nIn order to cope with an imbalance of (3), I removed same images.\nBut sampling images is not enough to balance the classes, so I introduced class weight.
", "rawMarkdown": "Congrats to all the winners! and I'd like to thank Google and Kaggle for this interesting competition. \n\nThis is My brief summary.\n\n# Model 1: object detection (yolo)\nI used yolo for object detection with following modification.\n\n- Removed confidence term from loss function\n- Added Class weight\n- Class masking to forces only on labeled classes \n But, did not mask the area where BB exists (BBs does not exist on same area) \n But, masked for child class (BBs can exist on same area for child classes) \n\n# Model 2: visual relationship (InceptionResNetV 2)\n \nRelation data(challenge-2018-train-vrd.csv) is mapped to all combinations of human labeled BBs(challenge-2018-train-vrd-bbox.csv) and the data is used as training data of model2.\n\n\n\n\n### 2-1: relation 'is'\nI made training data by mapping the ground truth BBOX(challenge-2018-train-vrd-bbox.csv) to ground truth relational data(challenge-2018-train-vrd.csv), giving a target. \n\n- Objective: Material (wooden, plastic, ..., None). \n- Features: Cropped image, BBOX class, BB position / size etc.\n\n\nExample:\t\n\nbbox data:\n\nimage1,bbox1,Man\n image1,bbox2,Guiter\n image1,bbox3,Chair\n\n\n\nRelationship data:\n\n
image1, bbox2, bbox2, Guitar, Wooden, is\n image1, bbox3, bbox3, Chair, Wooden, is\n image1, bbox1, bbox2, Man, Guitar, hold\n\n\nthen, training data would be\n\n
image1,bbox1,bbox1, Man,None,is \n image1,bbox2,bbox2, Guiter,Wooden,is\n image1,bbox3,bbox3, Chair ,Wooden,is \n\n here, target is None if the data is not in Relationship data.\n\n\n### 2-2: Triplet Relationships \n\nFirst, I created all pairs of BBOX in the same image with some filter.\nThen, I made training data by mapping the pairs to ground truth relational data, giving a target.\n\n- Objective: Relationship (at, on, ..., None). \n- Features: Cropped image including two BBOX with box line, LabelName1, LabelName2, XCenter1, YCenter1, XCenter2, YCenter2, Size1, Size2, Aspect1, Aspect2, XCenterDiff, YCenterDiff, CenterDiff, XCenter, YCenter, IOU\n\nExample:\nbbox data :\n\n
image1,bbox1,Man\n image1,bbox2,Guiter\n image1,bbox3,Chair\n\n\nRelationship data:\n\n
image1,bbox1,bbox2,Man,Guiter,hold\n\n\n\nThen training data would be \n\n
image1,bbox1,bbox2,Man,Guiter,hold\n image1,bbox1,bbox3,Man,Chair,None\n image1,bbox2,bbox1,Guiter,Man,None\n image1,bbox2,bbox3,Guiter,Chair,None\n image1,bbox3,bbox1,Chair,Man,None\n image1,bbox3,bbox2,Chair,Guiter,None\n\n here, target is None if the data is not in Relationship data.\n\n\n\n\n\n\n# Model 3: Score Prediction (Light GBM)\nRelation data(challenge-2018-train-vrd.csv) is mapped to all combinations of predicted BBs(output of model1), and the data is used as training data of model3.\n\n- Features: model1 output(BBOX classes, BBOX scores, BB positions / sizes etc.), and model2 output(Relationship and its probability)\n- Objective: whether the prediction of model2 is true or not\n\nModel2 and model3 can be merged to one NN model.\n\n# About yolo object function\n\nThe data set of this competition has the following three characteristics.\n\n## (1), every images are not checked for all classes.\nFor example, there are cases where cats are not checked even if there is a cat in the image.\nThis causes false penalty if the model detects a cat during learning.\n\n## (2), each object has a parent-child relationship.\nFor example, cat class is a child class of the Animal class, so a BBOX of cat is also a BBOX of Animal.\n\n## (3), the 500 classes are not balanced.\n\nTo handle these better, I first deleted the confidence term from YOLO's objective function.\nBy deleting this term, class probability will be responsible for confidence.\nOnce class probability also have confidence role, this make it possible to mask unchecked classes.\nBy masking the cat class in the example above it becomes possible to eliminate the penalty when cat is detected by the model.\n\nHowever, It is unlikely that different classes of boxes will be in the same area(same place, and same size, and same aspect).\nSo, I did not mask for the same area where some bbox exists.\n\nBut, there is an exception to this, as bbox of child class can exist in the same area(Animal can be a cat as well).\nSo I masked the child class of the corresponding BBOX.\n\nIn order to cope with an imbalance of (3), I removed same images.\nBut sampling images is not enough to balance the classes, so I introduced class weight.", "votes": null }, { "id": "379323", "postDate": "08/31/2018 07:01:18", "content": "
Congrats!
", "rawMarkdown": "Congrats!", "votes": null }, { "id": "379328", "postDate": "08/31/2018 07:11:11", "content": "Thank you, Ahmet!
", "rawMarkdown": "Thank you, Ahmet!", "votes": null }, { "id": "379475", "postDate": "08/31/2018 12:14:10", "content": "Congrats tito, great job! Thank you for the write up - very interesting read.
\n\nCould I please ask you how are you using the class weight in the detector? Are you using it to increase the weight of infrequent classes or in some other way?
\n\nAlso, could I please ask you what was your reasoning behind removing the confidence score from the loss?
\n\nThank you very much again for the write up - much appreciated.
", "rawMarkdown": "Congrats tito, great job! Thank you for the write up - very interesting read.\n\nCould I please ask you how are you using the class weight in the detector? Are you using it to increase the weight of infrequent classes or in some other way?\n\nAlso, could I please ask you what was your reasoning behind removing the confidence score from the loss?\n\nThank you very much again for the write up - much appreciated.", "votes": null }, { "id": "379485", "postDate": "08/31/2018 12:50:15", "content": "Thank you radek,
\n\nAs you mentioned, I used class weight to balance the infrequent classes.
\n\nI expected class probability takes over confidence by removing confidence term from loss function.\nThis (confidence for each class) made it possible to add class masking and class weight, etc to yolo.
", "rawMarkdown": "Thank you radek,\n\nAs you mentioned, I used class weight to balance the infrequent classes.\n\nI expected class probability takes over confidence by removing confidence term from loss function.\nThis (confidence for each class) made it possible to add class masking and class weight, etc to yolo.", "votes": null }, { "id": "379981", "postDate": "09/01/2018 11:33:19", "content": "Congratulations tito, and nice summary.
\n\nWill try to implement a model with your ideas.
\n\nAnd any plans to publish your code?
\n\nAll the best for the future.
", "rawMarkdown": "Congratulations tito, and nice summary. \n\nWill try to implement a model with your ideas. \n\nAnd any plans to publish your code?\n\nAll the best for the future.", "votes": null }, { "id": "380066", "postDate": "09/01/2018 15:32:52", "content": "Hi Nazim,
\n\nI'm not sure whether I'll publish my code yet.\nAnyway, I'll let you know here if I publish it.
", "rawMarkdown": "Hi Nazim,\n\nI'm not sure whether I'll publish my code yet.\nAnyway, I'll let you know here if I publish it.", "votes": null }, { "id": "380288", "postDate": "09/02/2018 08:25:37", "content": "Congratulations tito! Thank you for the writeup. I wonder if you could further explain how you combined the cropped image plus the other non-image features (labelname1 etc) into the 2-2 model?
", "rawMarkdown": "Congratulations tito! Thank you for the writeup. I wonder if you could further explain how you combined the cropped image plus the other non-image features (labelname1 etc) into the 2-2 model?", "votes": null }, { "id": "380297", "postDate": "09/02/2018 09:00:46", "content": "Hi Tim H,
\n\nI added Embeded categorical features and numerical features on the top of CNN as dens layer.\nI used following code for this.
\n\nBASE_MODEL='InceptionResNetV2'\ncategorical = ['name_1', 'name_2']\nnumerical = ['XCenter1','YCenter1','XCenter2','YCenter2','Size1','Size2','Aspect1','Aspect2','XCenterDiff','YCenterDiff','CenterDiff','XCenter','YCenter','IOU']\n\ndef mk_model():\n if BASE_MODEL=='VGG16':\n base_model = VGG16(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n elif BASE_MODEL=='InceptionV3':\n base_model=InceptionV3(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n elif BASE_MODEL=='Xception':\n base_model=Xception(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n elif BASE_MODEL=='InceptionResNetV2':\n base_model=InceptionResNetV2(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n input_list = []\n x1=base_model.output\n x1=GlobalAveragePooling2D()(x1)\n input_list.append(base_model.input)\n\n emb_list = []\n for cat in categorical:\n inpt = Input(shape=[1], name = cat)\n input_list.append(inpt)\n embd = Embedding(len(bb_values), len(bb_values))(inpt)\n emb_list.append(embd)\n x2 = concatenate(emb_list)\n x2 = SpatialDropout1D(0.5)(x2)\n x2 = Flatten()(x2)\n x2 = Dense(512,activation='relu')(x2)\n x2 = BatchNormalization()(x2)\n\n inpt = Input((len(numerical),),name='numerical')\n input_list.append(inpt)\n x3 = Dense(512,activation='relu')(inpt)\n x3 = Dropout(0.5)(x3)\n x3 = BatchNormalization()(x3)\n\n inpt = Input((len(bb_values)*2,),name='confidence')\n input_list.append(inpt)\n x4 = Dense(512,activation='relu')(inpt)\n x4 = Dropout(0.5)(x4)\n x4 = BatchNormalization()(x4)\n\n x = concatenate([x1, x2, x3, x4])\n x=Dense(1024,activation='relu')(x)\n x = Dropout(0.2)(x)\n x = BatchNormalization()(x)\n x=Dense(1024,activation='relu')(x)\n x = BatchNormalization()(x)\n prediction=Dense(n_classes,activation='sigmoid')(x)\n\n model=Model(inputs=input_list,outputs=prediction)\n for layer in base_model.layers:\n layer.trainable=False\n return model, base_model\n", "rawMarkdown": "Hi Tim H,\n\nI added Embeded categorical features and numerical features on the top of CNN as dens layer.\nI used following code for this.\n\n
BASE_MODEL='InceptionResNetV2'\ncategorical = ['name_1', 'name_2']\nnumerical = ['XCenter1','YCenter1','XCenter2','YCenter2','Size1','Size2','Aspect1','Aspect2','XCenterDiff','YCenterDiff','CenterDiff','XCenter','YCenter','IOU']\n \ndef mk_model():\n if BASE_MODEL=='VGG16':\n base_model = VGG16(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n elif BASE_MODEL=='InceptionV3':\n base_model=InceptionV3(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n elif BASE_MODEL=='Xception':\n base_model=Xception(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n elif BASE_MODEL=='InceptionResNetV2':\n base_model=InceptionResNetV2(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n input_list = []\n x1=base_model.output\n x1=GlobalAveragePooling2D()(x1)\n input_list.append(base_model.input)\n \n emb_list = []\n for cat in categorical:\n inpt = Input(shape=[1], name = cat)\n input_list.append(inpt)\n embd = Embedding(len(bb_values), len(bb_values))(inpt)\n emb_list.append(embd)\n x2 = concatenate(emb_list)\n x2 = SpatialDropout1D(0.5)(x2)\n x2 = Flatten()(x2)\n x2 = Dense(512,activation='relu')(x2)\n x2 = BatchNormalization()(x2)\n\n inpt = Input((len(numerical),),name='numerical')\n input_list.append(inpt)\n x3 = Dense(512,activation='relu')(inpt)\n x3 = Dropout(0.5)(x3)\n x3 = BatchNormalization()(x3)\n \n inpt = Input((len(bb_values)*2,),name='confidence')\n input_list.append(inpt)\n x4 = Dense(512,activation='relu')(inpt)\n x4 = Dropout(0.5)(x4)\n x4 = BatchNormalization()(x4)\n\n x = concatenate([x1, x2, x3, x4])\n x=Dense(1024,activation='relu')(x)\n x = Dropout(0.2)(x)\n x = BatchNormalization()(x)\n x=Dense(1024,activation='relu')(x)\n x = BatchNormalization()(x)\n prediction=Dense(n_classes,activation='sigmoid')(x)\n\n model=Model(inputs=input_list,outputs=prediction)\n for layer in base_model.layers:\n layer.trainable=False\n return model, base_model\n", "votes": null }, { "id": "380300", "postDate": "09/02/2018 09:09:33", "content": "
Thank you tito!
", "rawMarkdown": "Thank you tito!", "votes": null }, { "id": "399109", "postDate": "10/05/2018 08:15:53", "content": "Hi, Thank you for your reply!
\n\nI'm trying to understand your code.\nwhat does x1 represent? x1 = base_mode.output. So is it the classification prediction from the base_model, InceptionResNet or a feature map from the model?
\n\nx2 represents embedded representation of labels.\nx3 represents numerical inputs\nx4 are lists of bounding boxes for two objects detected.\nThen,\nyou concatenate them all to input into a model, right?
\n\nI think I'm confused why you append base_model.input into the input_list, which makes sense to me, but base_mode.output into x1. Do you mind explaining? Thank you!
\n\n----EDIT\nah, wait. I think I get it now. \nX1 to X4 are computed features from the layers using inputs like images, labels and bounding box annotations.
\n\nHow do you merge Model2.1 , 2.2 and 3?
", "rawMarkdown": "Hi, Thank you for your reply!\n\nI'm trying to understand your code.\nwhat does x1 represent? x1 = base_mode.output. So is it the classification prediction from the base_model, InceptionResNet or a feature map from the model?\n\nx2 represents embedded representation of labels.\nx3 represents numerical inputs\nx4 are lists of bounding boxes for two objects detected.\nThen,\nyou concatenate them all to input into a model, right? \n\nI think I'm confused why you append base_model.input into the input_list, which makes sense to me, but base_mode.output into x1. Do you mind explaining? Thank you!\n\n----EDIT\nah, wait. I think I get it now. \nX1 to X4 are computed features from the layers using inputs like images, labels and bounding box annotations.\n\nHow do you merge Model2.1 , 2.2 and 3?", "votes": null }, { "id": "399291", "postDate": "10/05/2018 14:52:38", "content": "Hi Kim,
\n\n\n\n\n----EDIT ah, wait. I think I get it now. X1 to X4 are computed features from the layers using inputs like images, labels and bounding box annotations.
\n
Yes, you understand it correctly.
\n\n\n\n\nHow do you merge Model2.1 , 2.2 and 3?
\n
Model3 is made for Model2.1 and Model2.2 respectively.
\n\nModel2.1 -> Model3.1 -> prediction for relation 'is'\nModel2.2 -> Model3.2 -> prediction for Triplet Relationships\nThen I just combined the predictions to one submission.
", "rawMarkdown": "Hi Kim,\n\n> ----EDIT ah, wait. I think I get it now. X1 to X4 are computed features from the layers using inputs like images, labels and bounding box annotations.\n\nYes, you understand it correctly.\n\n> How do you merge Model2.1 , 2.2 and 3?\n\nModel3 is made for Model2.1 and Model2.2 respectively.\n\n Model2.1 -> Model3.1 -> prediction for relation 'is'\n Model2.2 -> Model3.2 -> prediction for Triplet Relationships\n\nThen I just combined the predictions to one submission.", "votes": null }, { "id": "450420", "postDate": "01/04/2019 22:45:26", "content": "Could you please provide the python notebook for this competition?
", "rawMarkdown": "Could you please provide the python notebook for this competition?", "votes": null }, { "id": "450525", "postDate": "01/05/2019 07:22:22", "content": "Any update?
", "rawMarkdown": "Any update?", "votes": null } ], "comments": [ { "id": 379323, "author_name": "aerdem4", "author_url": "", "post_date": "08/31/2018 07:01:18", "content": "Congrats!
", "votes": null, "replies": [ { "id": 379328, "author_name": "its7171", "author_url": "", "post_date": "08/31/2018 07:11:11", "content": "Thank you, Ahmet!
", "votes": null, "replies": [] } ] }, { "id": 379475, "author_name": "radek1", "author_url": "", "post_date": "08/31/2018 12:14:10", "content": "Congrats tito, great job! Thank you for the write up - very interesting read.
\n\nCould I please ask you how are you using the class weight in the detector? Are you using it to increase the weight of infrequent classes or in some other way?
\n\nAlso, could I please ask you what was your reasoning behind removing the confidence score from the loss?
\n\nThank you very much again for the write up - much appreciated.
", "votes": null, "replies": [ { "id": 379485, "author_name": "its7171", "author_url": "", "post_date": "08/31/2018 12:50:15", "content": "Thank you radek,
\n\nAs you mentioned, I used class weight to balance the infrequent classes.
\n\nI expected class probability takes over confidence by removing confidence term from loss function.\nThis (confidence for each class) made it possible to add class masking and class weight, etc to yolo.
", "votes": null, "replies": [] } ] }, { "id": 379981, "author_name": "nazimgirach", "author_url": "", "post_date": "09/01/2018 11:33:19", "content": "Congratulations tito, and nice summary.
\n\nWill try to implement a model with your ideas.
\n\nAnd any plans to publish your code?
\n\nAll the best for the future.
", "votes": null, "replies": [ { "id": 380066, "author_name": "its7171", "author_url": "", "post_date": "09/01/2018 15:32:52", "content": "Hi Nazim,
\n\nI'm not sure whether I'll publish my code yet.\nAnyway, I'll let you know here if I publish it.
", "votes": null, "replies": [] }, { "id": 450525, "author_name": "nazimgirach", "author_url": "", "post_date": "01/05/2019 07:22:22", "content": "Any update?
", "votes": null, "replies": [] } ] }, { "id": 380288, "author_name": "timhartill", "author_url": "", "post_date": "09/02/2018 08:25:37", "content": "Congratulations tito! Thank you for the writeup. I wonder if you could further explain how you combined the cropped image plus the other non-image features (labelname1 etc) into the 2-2 model?
", "votes": null, "replies": [ { "id": 380297, "author_name": "its7171", "author_url": "", "post_date": "09/02/2018 09:00:46", "content": "Hi Tim H,
\n\nI added Embeded categorical features and numerical features on the top of CNN as dens layer.\nI used following code for this.
\n\nBASE_MODEL='InceptionResNetV2'\ncategorical = ['name_1', 'name_2']\nnumerical = ['XCenter1','YCenter1','XCenter2','YCenter2','Size1','Size2','Aspect1','Aspect2','XCenterDiff','YCenterDiff','CenterDiff','XCenter','YCenter','IOU']\n\ndef mk_model():\n if BASE_MODEL=='VGG16':\n base_model = VGG16(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n elif BASE_MODEL=='InceptionV3':\n base_model=InceptionV3(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n elif BASE_MODEL=='Xception':\n base_model=Xception(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n elif BASE_MODEL=='InceptionResNetV2':\n base_model=InceptionResNetV2(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n input_list = []\n x1=base_model.output\n x1=GlobalAveragePooling2D()(x1)\n input_list.append(base_model.input)\n\n emb_list = []\n for cat in categorical:\n inpt = Input(shape=[1], name = cat)\n input_list.append(inpt)\n embd = Embedding(len(bb_values), len(bb_values))(inpt)\n emb_list.append(embd)\n x2 = concatenate(emb_list)\n x2 = SpatialDropout1D(0.5)(x2)\n x2 = Flatten()(x2)\n x2 = Dense(512,activation='relu')(x2)\n x2 = BatchNormalization()(x2)\n\n inpt = Input((len(numerical),),name='numerical')\n input_list.append(inpt)\n x3 = Dense(512,activation='relu')(inpt)\n x3 = Dropout(0.5)(x3)\n x3 = BatchNormalization()(x3)\n\n inpt = Input((len(bb_values)*2,),name='confidence')\n input_list.append(inpt)\n x4 = Dense(512,activation='relu')(inpt)\n x4 = Dropout(0.5)(x4)\n x4 = BatchNormalization()(x4)\n\n x = concatenate([x1, x2, x3, x4])\n x=Dense(1024,activation='relu')(x)\n x = Dropout(0.2)(x)\n x = BatchNormalization()(x)\n x=Dense(1024,activation='relu')(x)\n x = BatchNormalization()(x)\n prediction=Dense(n_classes,activation='sigmoid')(x)\n\n model=Model(inputs=input_list,outputs=prediction)\n for layer in base_model.layers:\n layer.trainable=False\n return model, base_model\n", "votes": null, "replies": [] }, { "id": 380300, "author_name": "timhartill", "author_url": "", "post_date": "09/02/2018 09:09:33", "content": "
Thank you tito!
", "votes": null, "replies": [] }, { "id": 399109, "author_name": "lemonista", "author_url": "", "post_date": "10/05/2018 08:15:53", "content": "Hi, Thank you for your reply!
\n\nI'm trying to understand your code.\nwhat does x1 represent? x1 = base_mode.output. So is it the classification prediction from the base_model, InceptionResNet or a feature map from the model?
\n\nx2 represents embedded representation of labels.\nx3 represents numerical inputs\nx4 are lists of bounding boxes for two objects detected.\nThen,\nyou concatenate them all to input into a model, right?
\n\nI think I'm confused why you append base_model.input into the input_list, which makes sense to me, but base_mode.output into x1. Do you mind explaining? Thank you!
\n\n----EDIT\nah, wait. I think I get it now. \nX1 to X4 are computed features from the layers using inputs like images, labels and bounding box annotations.
\n\nHow do you merge Model2.1 , 2.2 and 3?
", "votes": null, "replies": [] }, { "id": 399291, "author_name": "its7171", "author_url": "", "post_date": "10/05/2018 14:52:38", "content": "Hi Kim,
\n\n\n\n\n----EDIT ah, wait. I think I get it now. X1 to X4 are computed features from the layers using inputs like images, labels and bounding box annotations.
\n
Yes, you understand it correctly.
\n\n\n\n\nHow do you merge Model2.1 , 2.2 and 3?
\n
Model3 is made for Model2.1 and Model2.2 respectively.
\n\nModel2.1 -> Model3.1 -> prediction for relation 'is'\nModel2.2 -> Model3.2 -> prediction for Triplet Relationships\nThen I just combined the predictions to one submission.
", "votes": null, "replies": [] } ] }, { "id": 450420, "author_name": "sriramr", "author_url": "", "post_date": "01/04/2019 22:45:26", "content": "Could you please provide the python notebook for this competition?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "379292": "Congrats to all the winners! and I'd like to thank Google and Kaggle for this interesting competition. \n\nThis is My brief summary.\n\n# Model 1: object detection (yolo)\nI used yolo for object detection with following modification.\n\n- Removed confidence term from loss function\n- Added Class weight\n- Class masking to forces only on labeled classes \n But, did not mask the area where BB exists (BBs does not exist on same area) \n But, masked for child class (BBs can exist on same area for child classes) \n\n# Model 2: visual relationship (InceptionResNetV 2)\n \nRelation data(challenge-2018-train-vrd.csv) is mapped to all combinations of human labeled BBs(challenge-2018-train-vrd-bbox.csv) and the data is used as training data of model2.\n\n\n\n\n### 2-1: relation 'is'\nI made training data by mapping the ground truth BBOX(challenge-2018-train-vrd-bbox.csv) to ground truth relational data(challenge-2018-train-vrd.csv), giving a target. \n\n- Objective: Material (wooden, plastic, ..., None). \n- Features: Cropped image, BBOX class, BB position / size etc.\n\n\nExample:\t\n\nbbox data:\n\nimage1,bbox1,Man\n image1,bbox2,Guiter\n image1,bbox3,Chair\n\n\n\nRelationship data:\n\n
image1, bbox2, bbox2, Guitar, Wooden, is\n image1, bbox3, bbox3, Chair, Wooden, is\n image1, bbox1, bbox2, Man, Guitar, hold\n\n\nthen, training data would be\n\n
image1,bbox1,bbox1, Man,None,is \n image1,bbox2,bbox2, Guiter,Wooden,is\n image1,bbox3,bbox3, Chair ,Wooden,is \n\n here, target is None if the data is not in Relationship data.\n\n\n### 2-2: Triplet Relationships \n\nFirst, I created all pairs of BBOX in the same image with some filter.\nThen, I made training data by mapping the pairs to ground truth relational data, giving a target.\n\n- Objective: Relationship (at, on, ..., None). \n- Features: Cropped image including two BBOX with box line, LabelName1, LabelName2, XCenter1, YCenter1, XCenter2, YCenter2, Size1, Size2, Aspect1, Aspect2, XCenterDiff, YCenterDiff, CenterDiff, XCenter, YCenter, IOU\n\nExample:\nbbox data :\n\n
image1,bbox1,Man\n image1,bbox2,Guiter\n image1,bbox3,Chair\n\n\nRelationship data:\n\n
image1,bbox1,bbox2,Man,Guiter,hold\n\n\n\nThen training data would be \n\n
image1,bbox1,bbox2,Man,Guiter,hold\n image1,bbox1,bbox3,Man,Chair,None\n image1,bbox2,bbox1,Guiter,Man,None\n image1,bbox2,bbox3,Guiter,Chair,None\n image1,bbox3,bbox1,Chair,Man,None\n image1,bbox3,bbox2,Chair,Guiter,None\n\n here, target is None if the data is not in Relationship data.\n\n\n\n\n\n\n# Model 3: Score Prediction (Light GBM)\nRelation data(challenge-2018-train-vrd.csv) is mapped to all combinations of predicted BBs(output of model1), and the data is used as training data of model3.\n\n- Features: model1 output(BBOX classes, BBOX scores, BB positions / sizes etc.), and model2 output(Relationship and its probability)\n- Objective: whether the prediction of model2 is true or not\n\nModel2 and model3 can be merged to one NN model.\n\n# About yolo object function\n\nThe data set of this competition has the following three characteristics.\n\n## (1), every images are not checked for all classes.\nFor example, there are cases where cats are not checked even if there is a cat in the image.\nThis causes false penalty if the model detects a cat during learning.\n\n## (2), each object has a parent-child relationship.\nFor example, cat class is a child class of the Animal class, so a BBOX of cat is also a BBOX of Animal.\n\n## (3), the 500 classes are not balanced.\n\nTo handle these better, I first deleted the confidence term from YOLO's objective function.\nBy deleting this term, class probability will be responsible for confidence.\nOnce class probability also have confidence role, this make it possible to mask unchecked classes.\nBy masking the cat class in the example above it becomes possible to eliminate the penalty when cat is detected by the model.\n\nHowever, It is unlikely that different classes of boxes will be in the same area(same place, and same size, and same aspect).\nSo, I did not mask for the same area where some bbox exists.\n\nBut, there is an exception to this, as bbox of child class can exist in the same area(Animal can be a cat as well).\nSo I masked the child class of the corresponding BBOX.\n\nIn order to cope with an imbalance of (3), I removed same images.\nBut sampling images is not enough to balance the classes, so I introduced class weight.", "379323": "Congrats!", "379328": "Thank you, Ahmet!", "379475": "Congrats tito, great job! Thank you for the write up - very interesting read.\n\nCould I please ask you how are you using the class weight in the detector? Are you using it to increase the weight of infrequent classes or in some other way?\n\nAlso, could I please ask you what was your reasoning behind removing the confidence score from the loss?\n\nThank you very much again for the write up - much appreciated.", "379485": "Thank you radek,\n\nAs you mentioned, I used class weight to balance the infrequent classes.\n\nI expected class probability takes over confidence by removing confidence term from loss function.\nThis (confidence for each class) made it possible to add class masking and class weight, etc to yolo.", "379981": "Congratulations tito, and nice summary. \n\nWill try to implement a model with your ideas. \n\nAnd any plans to publish your code?\n\nAll the best for the future.", "380066": "Hi Nazim,\n\nI'm not sure whether I'll publish my code yet.\nAnyway, I'll let you know here if I publish it.", "380288": "Congratulations tito! Thank you for the writeup. I wonder if you could further explain how you combined the cropped image plus the other non-image features (labelname1 etc) into the 2-2 model?", "380297": "Hi Tim H,\n\nI added Embeded categorical features and numerical features on the top of CNN as dens layer.\nI used following code for this.\n\n
BASE_MODEL='InceptionResNetV2'\ncategorical = ['name_1', 'name_2']\nnumerical = ['XCenter1','YCenter1','XCenter2','YCenter2','Size1','Size2','Aspect1','Aspect2','XCenterDiff','YCenterDiff','CenterDiff','XCenter','YCenter','IOU']\n \ndef mk_model():\n if BASE_MODEL=='VGG16':\n base_model = VGG16(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n elif BASE_MODEL=='InceptionV3':\n base_model=InceptionV3(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n elif BASE_MODEL=='Xception':\n base_model=Xception(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n elif BASE_MODEL=='InceptionResNetV2':\n base_model=InceptionResNetV2(include_top=False, weights='imagenet', input_tensor=Input(shape=(224,224,3)))\n input_list = []\n x1=base_model.output\n x1=GlobalAveragePooling2D()(x1)\n input_list.append(base_model.input)\n \n emb_list = []\n for cat in categorical:\n inpt = Input(shape=[1], name = cat)\n input_list.append(inpt)\n embd = Embedding(len(bb_values), len(bb_values))(inpt)\n emb_list.append(embd)\n x2 = concatenate(emb_list)\n x2 = SpatialDropout1D(0.5)(x2)\n x2 = Flatten()(x2)\n x2 = Dense(512,activation='relu')(x2)\n x2 = BatchNormalization()(x2)\n\n inpt = Input((len(numerical),),name='numerical')\n input_list.append(inpt)\n x3 = Dense(512,activation='relu')(inpt)\n x3 = Dropout(0.5)(x3)\n x3 = BatchNormalization()(x3)\n \n inpt = Input((len(bb_values)*2,),name='confidence')\n input_list.append(inpt)\n x4 = Dense(512,activation='relu')(inpt)\n x4 = Dropout(0.5)(x4)\n x4 = BatchNormalization()(x4)\n\n x = concatenate([x1, x2, x3, x4])\n x=Dense(1024,activation='relu')(x)\n x = Dropout(0.2)(x)\n x = BatchNormalization()(x)\n x=Dense(1024,activation='relu')(x)\n x = BatchNormalization()(x)\n prediction=Dense(n_classes,activation='sigmoid')(x)\n\n model=Model(inputs=input_list,outputs=prediction)\n for layer in base_model.layers:\n layer.trainable=False\n return model, base_model\n", "380300": "Thank you tito!", "399109": "Hi, Thank you for your reply!\n\nI'm trying to understand your code.\nwhat does x1 represent? x1 = base_mode.output. So is it the classification prediction from the base_model, InceptionResNet or a feature map from the model?\n\nx2 represents embedded representation of labels.\nx3 represents numerical inputs\nx4 are lists of bounding boxes for two objects detected.\nThen,\nyou concatenate them all to input into a model, right? \n\nI think I'm confused why you append base_model.input into the input_list, which makes sense to me, but base_mode.output into x1. Do you mind explaining? Thank you!\n\n----EDIT\nah, wait. I think I get it now. \nX1 to X4 are computed features from the layers using inputs like images, labels and bounding box annotations.\n\nHow do you merge Model2.1 , 2.2 and 3?", "399291": "Hi Kim,\n\n> ----EDIT ah, wait. I think I get it now. X1 to X4 are computed features from the layers using inputs like images, labels and bounding box annotations.\n\nYes, you understand it correctly.\n\n> How do you merge Model2.1 , 2.2 and 3?\n\nModel3 is made for Model2.1 and Model2.2 respectively.\n\n Model2.1 -> Model3.1 -> prediction for relation 'is'\n Model2.2 -> Model3.2 -> prediction for Triplet Relationships\n\nThen I just combined the predictions to one submission.", "450420": "Could you please provide the python notebook for this competition?", "450525": "Any update?" }, "source": "meta" }