What should I do if I want three different input sizes? I have three models with different input sizes, but preprocessing images to three input sizes is too time consuming(I have tried using multi-processing to crop and resize images then storing them in /kaggle/working directory, but it's too slow). I have an idea that cropping and resize jpg to a big size then using pooling layer to downsample input before feeding it to the model. Any good advice? Need help.
", "messages": [ { "id": 937475, "postDate": "2020-07-21T03:41:14.113Z", "content": "I would use the tfrecords that Chris has generated in several different sizes. He seems to have organized them well so you MIGHT have success reading three sizes but it's going to get messy. Way beyond my skill set for sure.
\n\nI have 3 inputs with the same size - it was a bit of a pain but I THINK that I am feeding three inputs of the same image with different augmentation to my model. That was a real hit and miss coding effort - will find out in the next day or two if it was worth the trouble.
\n\nFor sure at least one of the three images should be sized to match your input. That leaves only two images that are to be processed. I have seen that different resize statements can have huge differences in speed and memory usage.
\n\nMost of my work is on local PC with Ubuntu and I am resizing from 768x768. This is the resize that works the fastest, where b is the image size I want. I have resized to as small as 32x32.
\n\nimage = tf.image.resize(image, [b, b],\n method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)
The final submission has to inference on private test data, doesn't it? I don't understand why you are inferencing on external dataset generated by others. Or am I misunderstanding something?
", "rawMarkdown": "The final submission has to inference on private test data, doesn't it? I don't understand why you are inferencing on external dataset generated by others. Or am I misunderstanding something?" }, { "id": 938405, "postDate": "2020-07-21T14:03:37.060Z", "content": "Yes - you have misunderstanding.
\n\nJust try to use different resize method to see if your speed improves.
", "rawMarkdown": "Yes - you have misunderstanding.\n\nJust try to use different resize method to see if your speed improves." }, { "id": 938479, "postDate": "2020-07-21T15:04:15.890Z", "rawMarkdown": "", "isDeleted": true }, { "id": 938480, "postDate": "2020-07-21T15:04:15.890Z", "rawMarkdown": "", "isDeleted": true }, { "id": 938481, "postDate": "2020-07-21T15:04:15.890Z", "rawMarkdown": "", "isDeleted": true }, { "id": 938478, "postDate": "2020-07-21T15:04:15.890Z", "content": "Thanks for your reply, but I think I didn't state my problem clearly(English is not my mother language). My problem is about preprocessing jpeg images to different sizes in INFERENCE NOTEBOOK(whick I'm going to submit to generate final submission), not about resizing images for TRAINING NOTEBOOK. I already have 4 trained models, these 4 models are trained with 4 different image sizes. I'm worried the executing time of INFERENCE NOTEBOOK on private data subset will exceed the time limit of kaggle kernel, because I have to generate 4 sizes image for my TRAINED models and the resolution of image in private data subset maybe huge, like 6000x4000.
", "rawMarkdown": "Thanks for your reply, but I think I didn't state my problem clearly(English is not my mother language). My problem is about preprocessing jpeg images to different sizes in INFERENCE NOTEBOOK(whick I'm going to submit to generate final submission), not about resizing images for TRAINING NOTEBOOK. I already have 4 trained models, these 4 models are trained with 4 different image sizes. I'm worried the executing time of INFERENCE NOTEBOOK on private data subset will exceed the time limit of kaggle kernel, because I have to generate 4 sizes image for my TRAINED models and the resolution of image in private data subset maybe huge, like 6000x4000." }, { "id": 938575, "postDate": "2020-07-21T15:58:38.510Z", "content": "I was born in USA - not sure English is my native language but it's the only one I have.
\n\nThe private test set runs on the same records that the public set does - you control those. If you create tfrecords of size 32x32 for train and test (and add them as data) - that's what is used for the public and private LB scoring. If your inference runs fine and can generate a submission for the public test than the private test is going to work.
\n\nLike the popular kernel by Chris you can include images to the 4 sizes you want as a data set you have created. Since you don't want to shuffle it should not be a huge task to keep the four in sync.
\n\nAs a test - you could start with Chris kernel - strip out most of the code and put your inference code - using 4 of the different sizes that Chris provides in a dataset.
", "rawMarkdown": "I was born in USA - not sure English is my native language but it's the only one I have.\n\nThe private test set runs on the same records that the public set does - you control those. If you create tfrecords of size 32x32 for train and test (and add them as data) - that's what is used for the public and private LB scoring. If your inference runs fine and can generate a submission for the public test than the private test is going to work. \n\nLike the [popular kernel by Chris](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords/data) you can include images to the 4 sizes you want as a data set you have created. Since you don't want to shuffle it should not be a huge task to keep the four in sync.\n\nAs a test - you could start with Chris kernel - strip out most of the code and put your inference code - using 4 of the different sizes that Chris provides in a dataset.\n\n" }, { "id": 938692, "postDate": "2020-07-21T17:25:10.130Z", "content": "Thank you, I'm really worried that the executing time of my notebook will exceed kernel limit, because I plan to use multi models and TTA in my notebook😂
", "rawMarkdown": "Thank you, I'm really worried that the executing time of my notebook will exceed kernel limit, because I plan to use multi models and TTA in my notebook😂" } ] }, { "id": 937465, "postDate": "2020-07-21T03:34:04.090Z", "content": "Hi, I would create a different notebook (or experiment) for each image size, save the results in a private dataset and then ensamble them in a separate notebook.
", "rawMarkdown": "Hi, I would create a different notebook (or experiment) for each image size, save the results in a private dataset and then ensamble them in a separate notebook." }, { "id": 937455, "postDate": "2020-07-21T03:17:48.077Z", "content": "What should I do if I want three different input sizes? I have three models with different input sizes, but preprocessing images to three input sizes is too time consuming(I have tried using multi-processing to crop and resize images then storing them in /kaggle/working directory, but it's too slow). I have an idea that cropping and resize jpg to a big size then using pooling layer to downsample input before feeding it to the model. Any good advice? Need help.
", "rawMarkdown": "What should I do if I want three different input sizes? I have three models with different input sizes, but preprocessing images to three input sizes is too time consuming(I have tried using multi-processing to crop and resize images then storing them in /kaggle/working directory, but it's too slow). I have an idea that cropping and resize jpg to a big size then using pooling layer to downsample input before feeding it to the model. Any good advice? Need help." } ], "comments": [ { "id": 937475, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-07-21T03:41:14.113000", "content": "I would use the tfrecords that Chris has generated in several different sizes. He seems to have organized them well so you MIGHT have success reading three sizes but it's going to get messy. Way beyond my skill set for sure.
\n\nI have 3 inputs with the same size - it was a bit of a pain but I THINK that I am feeding three inputs of the same image with different augmentation to my model. That was a real hit and miss coding effort - will find out in the next day or two if it was worth the trouble.
\n\nFor sure at least one of the three images should be sized to match your input. That leaves only two images that are to be processed. I have seen that different resize statements can have huge differences in speed and memory usage.
\n\nMost of my work is on local PC with Ubuntu and I am resizing from 768x768. This is the resize that works the fastest, where b is the image size I want. I have resized to as small as 32x32.
\n\nimage = tf.image.resize(image, [b, b],\n method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)
The final submission has to inference on private test data, doesn't it? I don't understand why you are inferencing on external dataset generated by others. Or am I misunderstanding something?
", "votes": 0, "replies": [] }, { "id": 938405, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-07-21T14:03:37.060000", "content": "Yes - you have misunderstanding.
\n\nJust try to use different resize method to see if your speed improves.
", "votes": 0, "replies": [] }, { "id": 938479, "author_name": "", "author_url": "", "post_date": "2020-07-21T15:04:15.890000", "content": "", "votes": 0, "replies": [] }, { "id": 938480, "author_name": "", "author_url": "", "post_date": "2020-07-21T15:04:15.890000", "content": "", "votes": 0, "replies": [] }, { "id": 938481, "author_name": "", "author_url": "", "post_date": "2020-07-21T15:04:15.890000", "content": "", "votes": 0, "replies": [] }, { "id": 938478, "author_name": "Anthony Chan", "author_url": "", "post_date": "2020-07-21T15:04:15.890000", "content": "Thanks for your reply, but I think I didn't state my problem clearly(English is not my mother language). My problem is about preprocessing jpeg images to different sizes in INFERENCE NOTEBOOK(whick I'm going to submit to generate final submission), not about resizing images for TRAINING NOTEBOOK. I already have 4 trained models, these 4 models are trained with 4 different image sizes. I'm worried the executing time of INFERENCE NOTEBOOK on private data subset will exceed the time limit of kaggle kernel, because I have to generate 4 sizes image for my TRAINED models and the resolution of image in private data subset maybe huge, like 6000x4000.
", "votes": 0, "replies": [] }, { "id": 938575, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-07-21T15:58:38.510000", "content": "I was born in USA - not sure English is my native language but it's the only one I have.
\n\nThe private test set runs on the same records that the public set does - you control those. If you create tfrecords of size 32x32 for train and test (and add them as data) - that's what is used for the public and private LB scoring. If your inference runs fine and can generate a submission for the public test than the private test is going to work.
\n\nLike the popular kernel by Chris you can include images to the 4 sizes you want as a data set you have created. Since you don't want to shuffle it should not be a huge task to keep the four in sync.
\n\nAs a test - you could start with Chris kernel - strip out most of the code and put your inference code - using 4 of the different sizes that Chris provides in a dataset.
", "votes": 0, "replies": [] }, { "id": 938692, "author_name": "Anthony Chan", "author_url": "", "post_date": "2020-07-21T17:25:10.130000", "content": "Thank you, I'm really worried that the executing time of my notebook will exceed kernel limit, because I plan to use multi models and TTA in my notebook😂
", "votes": 0, "replies": [] } ] }, { "id": 937465, "author_name": "Santiago Viquez", "author_url": "", "post_date": "2020-07-21T03:34:04.090000", "content": "Hi, I would create a different notebook (or experiment) for each image size, save the results in a private dataset and then ensamble them in a separate notebook.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "937475": "I would use the tfrecords that Chris has generated in several different sizes. He seems to have organized them well so you MIGHT have success reading three sizes but it's going to get messy. Way beyond my skill set for sure.\n\nI have 3 inputs with the same size - it was a bit of a pain but I THINK that I am feeding three inputs of the same image with different augmentation to my model. That was a real hit and miss coding effort - will find out in the next day or two if it was worth the trouble.\n\nFor sure at least one of the three images should be sized to match your input. That leaves only two images that are to be processed. I have seen that different resize statements can have huge differences in speed and memory usage.\n\nMost of my work is on local PC with Ubuntu and I am resizing from 768x768. This is the resize that works the fastest, where b is the image size I want. I have resized to as small as 32x32. \n\n` image = tf.image.resize(image, [b, b],\n method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)`", "937465": "Hi, I would create a different notebook (or experiment) for each image size, save the results in a private dataset and then ensamble them in a separate notebook.", "937455": "What should I do if I want three different input sizes? I have three models with different input sizes, but preprocessing images to three input sizes is too time consuming(I have tried using multi-processing to crop and resize images then storing them in /kaggle/working directory, but it's too slow). I have an idea that cropping and resize jpg to a big size then using pooling layer to downsample input before feeding it to the model. Any good advice? Need help." } }