Windows 10 + Python 3.5 + Keras 2.2 + Keras-RetinaNet [0.4.1]: \nhttps://github.com/fizyr/keras-retinanet
\n\nIn this task, I was strongly guided by my results and predictions from the task of Google AI Open Images - Object Detection Track (ODT) simply due to the fact that the \"ITEM\" classes are contained in the predictions from that task. Namely, 62 classes out of 500. It seems logical that the more accurately you know the positions of the objects you seek, the more accurately you can determine the type of interaction between them. Subsequently, I even trained a separate model for these 62 classes of objects using training boxes from ODT.
\n\nStatement of the task: two objects from the set to 62 classes are given on the image, as well as their interaction from 10 types:\nRELATIONSHIP = ['at', 'on', 'holds', 'plays', 'interacts_with', 'wears', 'inside_of', 'under', 'hits', 'is']\nIn this case the type ‘is’ has a separate property - it is defined for one object and means the type of material, I called its attribute of the object.
\n\nIn general, I split the problem into two independent parts, the Item + Attribute search and the second Item 1+ Item 2 + Relationship search.
\n\nI trained two Retina models. The first one try to find for all possible pairs of Item + Attribute present in Train. In total, there were 42 such pairs. That is, each class is for example \"Plastic chair\", \"Wooden chair\", \"Transparent cup, etc.\". The main problem is that for some pairs there are very few samples. For example, for a Wooden Mug, only 5 images.\nThis model alone gives on LB ~ 0.05550
\n\nThe second model looks only for the types of objects from the list\nLABELS_ATTR = ['(made of) Leather', '(made of Textile', 'Plastic', 'Transparent', 'Wooden']\nI wasn’t able to use this model to increase score. But it probably possible for some rare cases from previous model.
\n\nIn this case, I trained the Retina model for 9 classes, which is looking for the fact of having some kind of relationship.
\n\nRELATIONSHIP = ['at', 'on', 'holds', 'plays', 'interacts_with', 'wears', 'inside_of', 'under', 'hits']
\n\nModel tries to predict minimal rectangle containing two items forming some kind of relationship
\n\nFor the prediction of Item1 and Item2, I took the best model from the ODT task.\nTo find the answer itself, a heuristic was invented. All found boxes for item1, item2 and relation were cut off at a certain threshold. Then there was a full search was searched for IOU between the boxes, if the Relation Box had an intersection with Item1 and Item2 and these Items are part of the allowed ones, then answer was added with confidence_score = (w1 * Conf_Item1 + w2 * Conf_Item2 + w3 * IOU1 + w4 * IOU2 + w5 * Conf_Relation) / (w1 + w2 + w3 + w4 + w5). The values for wi were selected through scipy.minimize.
\n\nThen on the received boxes the modified NMS algorithm was started, which worked on two boxes at once. This model (Item 1+ Item 2 + Relationship) independently gave the score 0.15330 on LB.
\n\nIn addition to the heuristics, I also prepared a LightGBM model with an AUC metric that tried to predict whether the predicted rectangles are valid or not on the base of the label, coordinates of the rectangles, areas of rectangles etc. Thus I tried to automatically cut off some non-existent cases, for example, when the first object for ‘under’ relationship is above the second, etc. And on validation this model was really better than my heuristic, but on LB it was for some reason worse. I think this is due to a small number of images in the validation - just slightly more than 5K or with some kind of error in the code. The best score was ~0.12 on LB.
\n\nDefinitely here you have to dig in the direction of second-level models. Ideally, split the whole train set with KFold and use it to generate a variety of OOF features, for example rectangles from different Object Detectors, etc.
\n\nIn general, the validation in this task has always been much more optimistic for me, but in general It worked in the right direction (improvement of validation improved LB). For example, validation for my best submission:\nRel mAP 0.263953 Rel Recall 0.619921 Phrases mAP 0.342934 Pred score: 0.366739\nLB: 0.20109 - almost two times lower.
\n\nWindows 10 + Python 3.5 + Keras 2.2 + Keras-RetinaNet [0.4.1]: \nhttps://github.com/fizyr/keras-retinanet
\n\nIn this task, I was strongly guided by my results and predictions from the task of Google AI Open Images - Object Detection Track (ODT) simply due to the fact that the \"ITEM\" classes are contained in the predictions from that task. Namely, 62 classes out of 500. It seems logical that the more accurately you know the positions of the objects you seek, the more accurately you can determine the type of interaction between them. Subsequently, I even trained a separate model for these 62 classes of objects using training boxes from ODT.
\n\nStatement of the task: two objects from the set to 62 classes are given on the image, as well as their interaction from 10 types:\nRELATIONSHIP = ['at', 'on', 'holds', 'plays', 'interacts_with', 'wears', 'inside_of', 'under', 'hits', 'is']\nIn this case the type ‘is’ has a separate property - it is defined for one object and means the type of material, I called its attribute of the object.
\n\nIn general, I split the problem into two independent parts, the Item + Attribute search and the second Item 1+ Item 2 + Relationship search.
\n\nI trained two Retina models. The first one try to find for all possible pairs of Item + Attribute present in Train. In total, there were 42 such pairs. That is, each class is for example \"Plastic chair\", \"Wooden chair\", \"Transparent cup, etc.\". The main problem is that for some pairs there are very few samples. For example, for a Wooden Mug, only 5 images.\nThis model alone gives on LB ~ 0.05550
\n\nThe second model looks only for the types of objects from the list\nLABELS_ATTR = ['(made of) Leather', '(made of Textile', 'Plastic', 'Transparent', 'Wooden']\nI wasn’t able to use this model to increase score. But it probably possible for some rare cases from previous model.
\n\nIn this case, I trained the Retina model for 9 classes, which is looking for the fact of having some kind of relationship.
\n\nRELATIONSHIP = ['at', 'on', 'holds', 'plays', 'interacts_with', 'wears', 'inside_of', 'under', 'hits']
\n\nModel tries to predict minimal rectangle containing two items forming some kind of relationship
\n\nFor the prediction of Item1 and Item2, I took the best model from the ODT task.\nTo find the answer itself, a heuristic was invented. All found boxes for item1, item2 and relation were cut off at a certain threshold. Then there was a full search was searched for IOU between the boxes, if the Relation Box had an intersection with Item1 and Item2 and these Items are part of the allowed ones, then answer was added with confidence_score = (w1 * Conf_Item1 + w2 * Conf_Item2 + w3 * IOU1 + w4 * IOU2 + w5 * Conf_Relation) / (w1 + w2 + w3 + w4 + w5). The values for wi were selected through scipy.minimize.
\n\nThen on the received boxes the modified NMS algorithm was started, which worked on two boxes at once. This model (Item 1+ Item 2 + Relationship) independently gave the score 0.15330 on LB.
\n\nIn addition to the heuristics, I also prepared a LightGBM model with an AUC metric that tried to predict whether the predicted rectangles are valid or not on the base of the label, coordinates of the rectangles, areas of rectangles etc. Thus I tried to automatically cut off some non-existent cases, for example, when the first object for ‘under’ relationship is above the second, etc. And on validation this model was really better than my heuristic, but on LB it was for some reason worse. I think this is due to a small number of images in the validation - just slightly more than 5K or with some kind of error in the code. The best score was ~0.12 on LB.
\n\nDefinitely here you have to dig in the direction of second-level models. Ideally, split the whole train set with KFold and use it to generate a variety of OOF features, for example rectangles from different Object Detectors, etc.
\n\nIn general, the validation in this task has always been much more optimistic for me, but in general It worked in the right direction (improvement of validation improved LB). For example, validation for my best submission:\nRel mAP 0.263953 Rel Recall 0.619921 Phrases mAP 0.342934 Pred score: 0.366739\nLB: 0.20109 - almost two times lower.
\n\nWill you be publishing your code for this competition as well?
", "rawMarkdown": "Will you be publishing your code for this competition as well?", "votes": null } ], "comments": [ { "id": 584274, "author_name": "rohitmidha23", "author_url": "", "post_date": "07/25/2019 16:42:09", "content": "Will you be publishing your code for this competition as well?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "379150": "Software\n========\n\nWindows 10 + Python 3.5 + Keras 2.2 + Keras-RetinaNet [0.4.1]: \nhttps://github.com/fizyr/keras-retinanet\n\nMain approach\n=============\n\nIn this task, I was strongly guided by my results and predictions from the task of Google AI Open Images - Object Detection Track (ODT) simply due to the fact that the \"ITEM\" classes are contained in the predictions from that task. Namely, 62 classes out of 500. It seems logical that the more accurately you know the positions of the objects you seek, the more accurately you can determine the type of interaction between them. Subsequently, I even trained a separate model for these 62 classes of objects using training boxes from ODT.\n\nStatement of the task: two objects from the set to 62 classes are given on the image, as well as their interaction from 10 types:\nRELATIONSHIP = ['at', 'on', 'holds', 'plays', 'interacts_with', 'wears', 'inside_of', 'under', 'hits', 'is']\nIn this case the type ‘is’ has a separate property - it is defined for one object and means the type of material, I called its attribute of the object.\n\nIn general, I split the problem into two independent parts, the Item + Attribute search and the second Item 1+ Item 2 + Relationship search.\n\nPart 1 (Item + Attribute)\n=========================\n\nI trained two Retina models. The first one try to find for all possible pairs of Item + Attribute present in Train. In total, there were 42 such pairs. That is, each class is for example \"Plastic chair\", \"Wooden chair\", \"Transparent cup, etc.\". The main problem is that for some pairs there are very few samples. For example, for a Wooden Mug, only 5 images.\nThis model alone gives on LB ~ 0.05550\n\nThe second model looks only for the types of objects from the list\nLABELS_ATTR = ['(made of) Leather', '(made of Textile', 'Plastic', 'Transparent', 'Wooden']\nI wasn’t able to use this model to increase score. But it probably possible for some rare cases from previous model.\n\nPart 2 (Item 1+ Item 2 + Relationship)\n======================================\n\nIn this case, I trained the Retina model for 9 classes, which is looking for the fact of having some kind of relationship. \n\nRELATIONSHIP = ['at', 'on', 'holds', 'plays', 'interacts_with', 'wears', 'inside_of', 'under', 'hits']\n\nModel tries to predict minimal rectangle containing two items forming some kind of relationship\n\nFor the prediction of Item1 and Item2, I took the best model from the ODT task.\nTo find the answer itself, a heuristic was invented. All found boxes for item1, item2 and relation were cut off at a certain threshold. Then there was a full search was searched for IOU between the boxes, if the Relation Box had an intersection with Item1 and Item2 and these Items are part of the allowed ones, then answer was added with confidence_score = (w1 * Conf_Item1 + w2 * Conf_Item2 + w3 * IOU1 + w4 * IOU2 + w5 * Conf_Relation) / (w1 + w2 + w3 + w4 + w5). The values for wi were selected through scipy.minimize.\n\nThen on the received boxes the modified NMS algorithm was started, which worked on two boxes at once. This model (Item 1+ Item 2 + Relationship) independently gave the score 0.15330 on LB.\n\nFailed experiments\n==================\n\nIn addition to the heuristics, I also prepared a LightGBM model with an AUC metric that tried to predict whether the predicted rectangles are valid or not on the base of the label, coordinates of the rectangles, areas of rectangles etc. Thus I tried to automatically cut off some non-existent cases, for example, when the first object for ‘under’ relationship is above the second, etc. And on validation this model was really better than my heuristic, but on LB it was for some reason worse. I think this is due to a small number of images in the validation - just slightly more than 5K or with some kind of error in the code. The best score was ~0.12 on LB.\n\nDefinitely here you have to dig in the direction of second-level models. Ideally, split the whole train set with KFold and use it to generate a variety of OOF features, for example rectangles from different Object Detectors, etc.\n\nIn general, the validation in this task has always been much more optimistic for me, but in general It worked in the right direction (improvement of validation improved LB). For example, validation for my best submission:\nRel mAP 0.263953 Rel Recall 0.619921 Phrases mAP 0.342934 Pred score: 0.366739\nLB: 0.20109 - almost two times lower.\n\nProblems of dataset\n===================\n\n - A relatively small amount of data in the training set\n - Absence of validation and its difference from the validation from the ODT task\n - The “under” class is almost not represented in the training set (only 34 samples)", "584274": "Will you be publishing your code for this competition as well?" }, "source": "meta" }