Hi, I want to discuss on image features - to what extent they are useful.
\n\n<1> \nI tried image feature extraction, and I only extracted blurrness score - you can look an one of my kernals . Since it's very time-consuming, I ran multiple kernals at once, 100,000 for each kernal to save my time.
\n\nHowever, I found from LGB feature importance, image blurrness score shows quite high contribution, but not as much as image_top_1 or price features. Also adding image blurrness score itself did not improve the score that muuch for me.
\n\nI want to ask to what extent image feature helped, and if it helped how did you processed and used image features?
\n\n<2>
\nAbout image_top_1 feature, all I know is that it's some kind of classification result from Avito. However, I'm not sure how it is classified and actually could not thoroughly understand the feature - Is is categorial? or numerical feature? (Is it a classification or a score?) Also would there be any better ways to use the feature other than just using image_top_1 features as it is given?
Try this if you have time; it came out to be the most important image feature for me among the meta features; Its roughly for kp detection:
\n\ndef keyp(img):\ntry: \n img = images_path + str(img) + \".jpg\"\n img = cv2.imread(img,0)\n fast = cv2.FastFeatureDetector_create()\n\n # find and draw the keypoints\n kp = fast.detect(img,None)\n kp =len(kp)\n return kp\nexcept:\n return 0\n@Nooh, how long did your script take to get this feature? And what is your HW setup that you used if you do not mind sharing?
", "rawMarkdown": "@Nooh, how long did your script take to get this feature? And what is your HW setup that you used if you do not mind sharing?", "votes": 1 }, { "id": 347952, "postDate": "2018-06-25T19:00:42.403Z", "content": "Got an hp workstation machine 16 core ht, with 64g ram. I crunch the data on ssd and It took almost 15 hours for total 5-6 features for both train and test set. It's a windows machine so you know the lag. :/\nStill worth spending time. It improved both my lgb and xgb and eventually improved the ensemble.
", "rawMarkdown": "Got an hp workstation machine 16 core ht, with 64g ram. I crunch the data on ssd and It took almost 15 hours for total 5-6 features for both train and test set. It's a windows machine so you know the lag. :/\nStill worth spending time. It improved both my lgb and xgb and eventually improved the ensemble. ", "votes": 4 }, { "id": 347954, "postDate": "2018-06-25T19:07:53.233Z", "content": "btw i would love to public the image features data set in the spirit of sharing but still not sure if that will be fair or unfair for anyone. Scripts are nearly public or you can get it from Google easily, it all comes down to the processing power which frankly not everyone has.
", "rawMarkdown": "btw i would love to public the image features data set in the spirit of sharing but still not sure if that will be fair or unfair for anyone. Scripts are nearly public or you can get it from Google easily, it all comes down to the processing power which frankly not everyone has. ", "votes": 2 }, { "id": 347999, "postDate": "2018-06-25T21:13:12.330Z", "content": "Thanks @Nooh. It is too late for this contest but I was curious as to the time requirements.
\n\nAbout releasing the result it is up to you. When you mentioned sometime back that you would be releasing it, I was looking forward to it because I am one of those with a weak HW and I have already spent quite a bit on my cloud account. Since you did not I figured you changed your mind which is a god given right:-) Now I am just working on running my models 10Folds one last time and then run my stacker. Good Luck.
", "rawMarkdown": "Thanks @Nooh. It is too late for this contest but I was curious as to the time requirements. \n\nAbout releasing the result it is up to you. When you mentioned sometime back that you would be releasing it, I was looking forward to it because I am one of those with a weak HW and I have already spent quite a bit on my cloud account. Since you did not I figured you changed your mind which is a god given right:-) Now I am just working on running my models 10Folds one last time and then run my stacker. Good Luck.", "votes": 1 }, { "id": 348042, "postDate": "2018-06-26T00:51:11.173Z", "content": "Thanks for the suggestion! Sadly I'm not sure if I could successfully run this due to the weak hardware! But anyway thanks for sharing your knowledge, I appreciate a lot :)
", "rawMarkdown": "Thanks for the suggestion! Sadly I'm not sure if I could successfully run this due to the weak hardware! But anyway thanks for sharing your knowledge, I appreciate a lot :)", "votes": 1 }, { "id": 348095, "postDate": "2018-06-26T04:50:36.023Z", "content": "Ah! sad to hear that man. I'm comparatively new here and was afraid to upset other guys as the last competition became a chicken run at the end. It was the last week when I was confident about my image features being fruitful. But I think I thought too much there. :/ It wasn't a crime anyways.\nBest of luck to you bro!
", "rawMarkdown": "Ah! sad to hear that man. I'm comparatively new here and was afraid to upset other guys as the last competition became a chicken run at the end. It was the last week when I was confident about my image features being fruitful. But I think I thought too much there. :/ It wasn't a crime anyways.\nBest of luck to you bro!" }, { "id": 348096, "postDate": "2018-06-26T04:51:22.850Z", "content": "You'r welcome Ethan. Wish I could have shared it earlier.
", "rawMarkdown": "You'r welcome Ethan. Wish I could have shared it earlier." }, { "id": 348100, "postDate": "2018-06-26T05:09:55.707Z", "content": ":) For the sake of further exploration, do you have any idea of sharing the data even after the contest's over? It would be great if I could figure how could I use the image features, but sadly I think I won't be able to do that due to HW
", "rawMarkdown": ":) For the sake of further exploration, do you have any idea of sharing the data even after the contest's over? It would be great if I could figure how could I use the image features, but sadly I think I won't be able to do that due to HW " }, { "id": 348103, "postDate": "2018-06-26T05:26:32.230Z", "content": "Thanks for the KP code Nooh, using 20 cores and 20 threads ,I managed to obtain the features in 15 minutes
", "rawMarkdown": "Thanks for the KP code Nooh, using 20 cores and 20 threads ,I managed to obtain the features in 15 minutes", "votes": 2 }, { "id": 348146, "postDate": "2018-06-26T07:04:46.553Z", "content": "You should use the library multiprocessing from python and use Pool with the number of cpu cores:
\n\nfrom multiprocessing import Pool\nimport tqdm\ncpu_cores = 6 #this is in my case\np = Pool(cpu_cores)\nfeatures_train = list(tqdm.tqdm(p.imap(keyp, train_samples), total=len(train_samples)))\np.close()\np.join()\np.terminate()\nIt took 8 minutes for me
", "rawMarkdown": "You should use the library multiprocessing from python and use Pool with the number of cpu cores:\n\n \n\n from multiprocessing import Pool\n import tqdm\n cpu_cores = 6 #this is in my case\n p = Pool(cpu_cores)\n features_train = list(tqdm.tqdm(p.imap(keyp, train_samples), total=len(train_samples)))\n p.close()\n p.join()\n p.terminate()\n\nIt took 8 minutes for me", "votes": 5 }, { "id": 348149, "postDate": "2018-06-26T07:08:09.823Z", "content": "Sure thing buddy. will do that. @Ethan
", "rawMarkdown": "Sure thing buddy. will do that. @Ethan" }, { "id": 348150, "postDate": "2018-06-26T07:08:34.433Z", "content": "@Kin WoW! Good stuff man.
", "rawMarkdown": "@Kin WoW! Good stuff man. " }, { "id": 348151, "postDate": "2018-06-26T07:09:41.353Z", "content": "Damn! I'm bad in this area. Thanks for the share man! It will surely help things moving for people who still need the img features. @Eduardo ^
", "rawMarkdown": "Damn! I'm bad in this area. Thanks for the share man! It will surely help things moving for people who still need the img features. @Eduardo ^", "votes": 1 }, { "id": 348170, "postDate": "2018-06-26T07:40:33.970Z", "rawMarkdown": "", "isDeleted": true }, { "id": 348388, "postDate": "2018-06-26T16:32:37.770Z", "content": "Thank you for sharing! It took only one hour to run on kernel without parallel processing. Sadly it didn't bring me improvement, maybe I did something wrong.
", "rawMarkdown": "Thank you for sharing! It took only one hour to run on kernel without parallel processing. Sadly it didn't bring me improvement, maybe I did something wrong.", "isDeleted": true }, { "id": 348588, "postDate": "2018-06-27T01:28:55.460Z", "content": "Thank you for your suggestion!\nI would really wanna try, but not a proper hardware.... I will have to figure out some way from next time.\nThanks anyway!
", "rawMarkdown": "Thank you for your suggestion!\nI would really wanna try, but not a proper hardware.... I will have to figure out some way from next time.\nThanks anyway!\n" }, { "id": 348595, "postDate": "2018-06-27T01:45:10.430Z", "content": "From my new kernal here, I'm try the method Nooh suggested by diredtly working in zip files, without loading the file itself. (Welcomed to give any comments on this one!)
\n\nIt works better for the people with lower HW, and also this is the method I used for Image blurrness score extraction. Still, I'm not sure if this is the proper way since I'm a beginner and referred to many others' kernal and references. From this kernal I've extracted KP from train dataset and now extracting from test dataset too! I'll try applying this to my original work and see if it makes any improvements (even we've got only few more times though)
", "rawMarkdown": "From my new kernal [here][1], I'm try the method Nooh suggested by diredtly working in zip files, without loading the file itself. (Welcomed to give any comments on this one!)\n\nIt works better for the people with lower HW, and also this is the method I used for Image blurrness score extraction. Still, I'm not sure if this is the proper way since I'm a beginner and referred to many others' kernal and references. From this kernal I've extracted KP from train dataset and now extracting from test dataset too! I'll try applying this to my original work and see if it makes any improvements (even we've got only few more times though)\n\n\n\n [1]: https://www.kaggle.com/sukhyun9673/extracting-image-features-test", "votes": 1 }, { "id": 348850, "postDate": "2018-06-27T11:56:13.193Z", "content": "Thank you! I'll look into it!
", "rawMarkdown": "Thank you! I'll look into it!" } ] }, { "id": 346563, "postDate": "2018-06-22T00:29:00.807Z", "content": "Hi, I want to discuss on image features - to what extent they are useful.
\n\n<1> \nI tried image feature extraction, and I only extracted blurrness score - you can look an one of my kernals . Since it's very time-consuming, I ran multiple kernals at once, 100,000 for each kernal to save my time.
\n\nHowever, I found from LGB feature importance, image blurrness score shows quite high contribution, but not as much as image_top_1 or price features. Also adding image blurrness score itself did not improve the score that muuch for me.
\n\nI want to ask to what extent image feature helped, and if it helped how did you processed and used image features?
\n\n<2>
\nAbout image_top_1 feature, all I know is that it's some kind of classification result from Avito. However, I'm not sure how it is classified and actually could not thoroughly understand the feature - Is is categorial? or numerical feature? (Is it a classification or a score?) Also would there be any better ways to use the feature other than just using image_top_1 features as it is given?
Roughly how much time did it take to get blurrness score for 100000 images?
", "rawMarkdown": "Roughly how much time did it take to get blurrness score for 100000 images?", "replies": [ { "id": 347638, "postDate": "2018-06-24T23:15:04.387Z", "content": "About 6 hours!
", "rawMarkdown": "About 6 hours!" } ] }, { "id": 348145, "postDate": "2018-06-26T07:03:26.760Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 347781, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-25T11:26:50.540000", "content": "Try this if you have time; it came out to be the most important image feature for me among the meta features; Its roughly for kp detection:
\n\ndef keyp(img):\ntry: \n img = images_path + str(img) + \".jpg\"\n img = cv2.imread(img,0)\n fast = cv2.FastFeatureDetector_create()\n\n # find and draw the keypoints\n kp = fast.detect(img,None)\n kp =len(kp)\n return kp\nexcept:\n return 0\n@Nooh, how long did your script take to get this feature? And what is your HW setup that you used if you do not mind sharing?
", "votes": 1, "replies": [] }, { "id": 347952, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-25T19:00:42.403000", "content": "Got an hp workstation machine 16 core ht, with 64g ram. I crunch the data on ssd and It took almost 15 hours for total 5-6 features for both train and test set. It's a windows machine so you know the lag. :/\nStill worth spending time. It improved both my lgb and xgb and eventually improved the ensemble.
", "votes": 4, "replies": [] }, { "id": 347954, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-25T19:07:53.233000", "content": "btw i would love to public the image features data set in the spirit of sharing but still not sure if that will be fair or unfair for anyone. Scripts are nearly public or you can get it from Google easily, it all comes down to the processing power which frankly not everyone has.
", "votes": 2, "replies": [] }, { "id": 347999, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-06-25T21:13:12.330000", "content": "Thanks @Nooh. It is too late for this contest but I was curious as to the time requirements.
\n\nAbout releasing the result it is up to you. When you mentioned sometime back that you would be releasing it, I was looking forward to it because I am one of those with a weak HW and I have already spent quite a bit on my cloud account. Since you did not I figured you changed your mind which is a god given right:-) Now I am just working on running my models 10Folds one last time and then run my stacker. Good Luck.
", "votes": 1, "replies": [] }, { "id": 348042, "author_name": "Ethan Sukhyun Hong", "author_url": "", "post_date": "2018-06-26T00:51:11.173000", "content": "Thanks for the suggestion! Sadly I'm not sure if I could successfully run this due to the weak hardware! But anyway thanks for sharing your knowledge, I appreciate a lot :)
", "votes": 1, "replies": [] }, { "id": 348095, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-26T04:50:36.023000", "content": "Ah! sad to hear that man. I'm comparatively new here and was afraid to upset other guys as the last competition became a chicken run at the end. It was the last week when I was confident about my image features being fruitful. But I think I thought too much there. :/ It wasn't a crime anyways.\nBest of luck to you bro!
", "votes": 0, "replies": [] }, { "id": 348096, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-26T04:51:22.850000", "content": "You'r welcome Ethan. Wish I could have shared it earlier.
", "votes": 0, "replies": [] }, { "id": 348100, "author_name": "Ethan Sukhyun Hong", "author_url": "", "post_date": "2018-06-26T05:09:55.707000", "content": ":) For the sake of further exploration, do you have any idea of sharing the data even after the contest's over? It would be great if I could figure how could I use the image features, but sadly I think I won't be able to do that due to HW
", "votes": 0, "replies": [] }, { "id": 348103, "author_name": "Ee Kin Chin", "author_url": "", "post_date": "2018-06-26T05:26:32.230000", "content": "Thanks for the KP code Nooh, using 20 cores and 20 threads ,I managed to obtain the features in 15 minutes
", "votes": 2, "replies": [] }, { "id": 348146, "author_name": "EduMI95", "author_url": "", "post_date": "2018-06-26T07:04:46.553000", "content": "You should use the library multiprocessing from python and use Pool with the number of cpu cores:
\n\nfrom multiprocessing import Pool\nimport tqdm\ncpu_cores = 6 #this is in my case\np = Pool(cpu_cores)\nfeatures_train = list(tqdm.tqdm(p.imap(keyp, train_samples), total=len(train_samples)))\np.close()\np.join()\np.terminate()\nIt took 8 minutes for me
", "votes": 5, "replies": [] }, { "id": 348149, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-26T07:08:09.823000", "content": "Sure thing buddy. will do that. @Ethan
", "votes": 0, "replies": [] }, { "id": 348150, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-26T07:08:34.433000", "content": "@Kin WoW! Good stuff man.
", "votes": 0, "replies": [] }, { "id": 348151, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-26T07:09:41.353000", "content": "Damn! I'm bad in this area. Thanks for the share man! It will surely help things moving for people who still need the img features. @Eduardo ^
", "votes": 1, "replies": [] }, { "id": 348170, "author_name": "", "author_url": "", "post_date": "2018-06-26T07:40:33.970000", "content": "", "votes": 0, "replies": [] }, { "id": 348388, "author_name": "", "author_url": "", "post_date": "2018-06-26T16:32:37.770000", "content": "Thank you for sharing! It took only one hour to run on kernel without parallel processing. Sadly it didn't bring me improvement, maybe I did something wrong.
", "votes": 0, "replies": [] }, { "id": 348588, "author_name": "Eric Park", "author_url": "", "post_date": "2018-06-27T01:28:55.460000", "content": "Thank you for your suggestion!\nI would really wanna try, but not a proper hardware.... I will have to figure out some way from next time.\nThanks anyway!
", "votes": 0, "replies": [] }, { "id": 348595, "author_name": "Ethan Sukhyun Hong", "author_url": "", "post_date": "2018-06-27T01:45:10.430000", "content": "From my new kernal here, I'm try the method Nooh suggested by diredtly working in zip files, without loading the file itself. (Welcomed to give any comments on this one!)
\n\nIt works better for the people with lower HW, and also this is the method I used for Image blurrness score extraction. Still, I'm not sure if this is the proper way since I'm a beginner and referred to many others' kernal and references. From this kernal I've extracted KP from train dataset and now extracting from test dataset too! I'll try applying this to my original work and see if it makes any improvements (even we've got only few more times though)
", "votes": 1, "replies": [] }, { "id": 348850, "author_name": "Eric Park", "author_url": "", "post_date": "2018-06-27T11:56:13.193000", "content": "Thank you! I'll look into it!
", "votes": 0, "replies": [] } ] }, { "id": 347487, "author_name": "Rajesh Shreedhar", "author_url": "", "post_date": "2018-06-24T13:52:27.043000", "content": "Roughly how much time did it take to get blurrness score for 100000 images?
", "votes": 0, "replies": [ { "id": 347638, "author_name": "Ethan Sukhyun Hong", "author_url": "", "post_date": "2018-06-24T23:15:04.387000", "content": "About 6 hours!
", "votes": 0, "replies": [] } ] }, { "id": 348145, "author_name": "", "author_url": "", "post_date": "2018-06-26T07:03:26.760000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "347781": "Try this if you have time; it came out to be the most important image feature for me among the meta features; Its roughly for kp detection:\n\n def keyp(img):\n try: \n img = images_path + str(img) + \".jpg\"\n img = cv2.imread(img,0)\n fast = cv2.FastFeatureDetector_create()\n\n # find and draw the keypoints\n kp = fast.detect(img,None)\n kp =len(kp)\n return kp\n except:\n return 0\n\n ", "346563": "Hi, I want to discuss on image features - to what extent they are useful. \n\n<1> \nI tried image feature extraction, and I only extracted blurrness score - you can look an one of [my kernals][1] . Since it's very time-consuming, I ran multiple kernals at once, 100,000 for each kernal to save my time.\n\nHowever, I found from LGB feature importance, image blurrness score shows quite high contribution, but not as much as image_top_1 or price features. Also adding image blurrness score itself did not improve the score that muuch for me. \n\nI want to ask to what extent image feature helped, and if it helped how did you processed and used image features?\n\n<2> \nAbout image_top_1 feature, all I know is that it's some kind of classification result from Avito. However, I'm not sure how it is classified and actually could not thoroughly understand the feature - Is is categorial? or numerical feature? (Is it a classification or a score?) Also would there be any better ways to use the feature other than just using image_top_1 features as it is given?\n\n [1]: https://www.kaggle.com/sukhyun9673/image-processing-600000-to-750000", "347487": "Roughly how much time did it take to get blurrness score for 100000 images?", "348145": "" } }