\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1303569%2F13b47c4858b34bae505ac82662d28df4%2FSubmission%20Error%20Screenshot.jpg?generation=1682478808560391&alt=media\")
The former submision is alright, but got time out after added this line:
\nconverter.optimizations = [tf.lite.Optimize.DEFAULT]
\n😧
tflite quantized models are optimized for ARM devices. For x86 CPU's, FP32 tflite model is faster than INT8 quantized tflite.
\nhttps://github.com/tensorflow/tensorflow/issues/21698#issuecomment-414764709
", "rawMarkdown": "tflite quantized models are optimized for ARM devices. For x86 CPU's, FP32 tflite model is faster than INT8 quantized tflite.\n\nhttps://github.com/tensorflow/tensorflow/issues/21698#issuecomment-414764709", "votes": 9 }, { "id": 2199612, "postDate": "2023-03-27T21:11:58.913Z", "content": "float16 quantization tends to works well with no speed or accuracy loss.
\n\n", "rawMarkdown": "float16 quantization tends to works well with no speed or accuracy loss.\n>keras_model_converter.optimizations = [tf.lite.Optimize.DEFAULT]\nkeras_model_converter.target_spec.supported_types = [tf.float16]\ntflite_model = keras_model_converter.convert() ", "votes": 6, "replies": [ { "id": 2199648, "postDate": "2023-03-27T22:41:41.800Z", "content": "keras_model_converter.optimizations = [tf.lite.Optimize.DEFAULT]
\n
\n keras_model_converter.target_spec.supported_types = [tf.float16]
\n tflite_model = keras_model_converter.convert()
for me it drops performance by ~10% and accuracy by 1-3%
\nEven with converting dataset. But maybe it will help someone
\nnum_calibration_steps = \n ():\n i (num_calibration_steps):\n [load_relevant_data_subset(BASE_DIR + train_df.loc[i, ])]\n\nkeras_model_converter.optimizations = [tf.lite.Optimize.DEFAULT]\nkeras_model_converter.representative_dataset = representative_dataset_gen\nkeras_model_converter.target_spec.supported_types = [tf.float16]\ntflite_model = keras_model_converter.convert()\nI don't think you need any calibration for fp16, for typical networks, model weight range should be well within the fp16 range.
\nfp16 conversion has worked well for past competitions in kaggle and during my work too (using pytorch), and you shouldn't need any calibrations.
\nWhat will happen if you exclude keras_model_converter.representative_dataset = representative_dataset_gen?
With my model and fp16 - absolutely nothing. It helps me a little with int8, but integer quantization performs much worse anyway (as @samfc10 have mentioned below).
\nI just wanted to leave it here because maybe it will improve someone's model that is okay with quantization.
", "rawMarkdown": "With my model and fp16 - absolutely nothing. It helps me a little with int8, but integer quantization performs much worse anyway (as @samfc10 have mentioned below).\n\nI just wanted to leave it here because maybe it will improve someone's model that is okay with quantization." } ] }, { "id": 2200439, "postDate": "2023-03-28T14:58:28.117Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2218419, "postDate": "2023-04-11T16:38:52.007Z", "content": "In my case, fp16 quantization causes severe accuracy drop (CV: 0.77955 -> 0.00421).
\nOn the other hand, int8 quantization works fine (CV: 0.77955 -> 0.77948).
converter = tf.lite.TFLiteConverter.from_keras_model(model)\n QUANTIZE [, ]:\n converter.optimizations = [tf.lite.Optimize.DEFAULT]\n QUANTIZE == :\n converter.target_spec.supported_types = [tf.float16]\n tflite_model = converter.convert()\nThank you for sharing!
\nI wonder how did you covert fp32 into int8?
@heoynwoo I used the same code shared above (you should set QUANTIZE = \"int8\").
\nTo cralify, int8 quantization result in about 20% slower inference time.
\nlocal evaluation time is:
\nint8 quantization turned out to be seriously slower in the Kaggle's evaluation environment.
\nit took far more longer time compared to local evaluation.
below is a comparison of the actual submission time:
\n| quantization | \nlocal eval time (sec) | \nestimated submission time (min) | \nactual submission time (min) | \n
|---|---|---|---|
| tf.float32 | \n167 | \n- | \n44 | \n
| tf.int8 | \n257 | \n67.7 | \n93 (timeout errror) | \n
Thank you! how did you measure actual submission time??
", "rawMarkdown": "Thank you! how did you measure actual submission time??" }, { "id": 2218894, "postDate": "2023-04-12T05:47:48.570Z", "content": "@heoynwoo You can poll the status of submission using Kaggle API.
\nVarious public notebooks exists:
\ne.g. https://www.kaggle.com/code/yasufuminakama/fb3-submission-time?scriptVersionId=104649325
The former submision is alright, but got time out after added this line:
\nconverter.optimizations = [tf.lite.Optimize.DEFAULT]
\n😧
In my experience, it all depends on the model. There are models that compress well and work well. Quantization can help in some cases, plus this is only the basic version, there are more complex options.
", "rawMarkdown": "In my experience, it all depends on the model. There are models that compress well and work well. Quantization can help in some cases, plus this is only the basic version, there are more complex options.", "votes": 2 }, { "id": 2235463, "postDate": "2023-04-26T04:23:33.440Z", "content": "In my specific case, submitting without any quantization results in normal result in about 45-60 mins while default (int8 quantization) optimization gave an output error in my model. Screenshot below:
\n
tflite quantized models are optimized for ARM devices. For x86 CPU's, FP32 tflite model is faster than INT8 quantized tflite.
\nhttps://github.com/tensorflow/tensorflow/issues/21698#issuecomment-414764709
", "votes": 9, "replies": [] }, { "id": 2199612, "author_name": "arutema47", "author_url": "", "post_date": "2023-03-27T21:11:58.913000", "content": "float16 quantization tends to works well with no speed or accuracy loss.
\n\n", "votes": 6, "replies": [ { "id": 2199648, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-03-27T22:41:41.800000", "content": "keras_model_converter.optimizations = [tf.lite.Optimize.DEFAULT]
\n
\n keras_model_converter.target_spec.supported_types = [tf.float16]
\n tflite_model = keras_model_converter.convert()
for me it drops performance by ~10% and accuracy by 1-3%
\nEven with converting dataset. But maybe it will help someone
\nnum_calibration_steps = \n ():\n i (num_calibration_steps):\n [load_relevant_data_subset(BASE_DIR + train_df.loc[i, ])]\n\nkeras_model_converter.optimizations = [tf.lite.Optimize.DEFAULT]\nkeras_model_converter.representative_dataset = representative_dataset_gen\nkeras_model_converter.target_spec.supported_types = [tf.float16]\ntflite_model = keras_model_converter.convert()\nI don't think you need any calibration for fp16, for typical networks, model weight range should be well within the fp16 range.
\nfp16 conversion has worked well for past competitions in kaggle and during my work too (using pytorch), and you shouldn't need any calibrations.
\nWhat will happen if you exclude keras_model_converter.representative_dataset = representative_dataset_gen?
With my model and fp16 - absolutely nothing. It helps me a little with int8, but integer quantization performs much worse anyway (as @samfc10 have mentioned below).
\nI just wanted to leave it here because maybe it will improve someone's model that is okay with quantization.
", "votes": 0, "replies": [] } ] }, { "id": 2200439, "author_name": "", "author_url": "", "post_date": "2023-03-28T14:58:28.117000", "content": "", "votes": 0, "replies": [] }, { "id": 2218419, "author_name": "Bilzard", "author_url": "", "post_date": "2023-04-11T16:38:52.007000", "content": "In my case, fp16 quantization causes severe accuracy drop (CV: 0.77955 -> 0.00421).
\nOn the other hand, int8 quantization works fine (CV: 0.77955 -> 0.77948).
converter = tf.lite.TFLiteConverter.from_keras_model(model)\n QUANTIZE [, ]:\n converter.optimizations = [tf.lite.Optimize.DEFAULT]\n QUANTIZE == :\n converter.target_spec.supported_types = [tf.float16]\n tflite_model = converter.convert()\nThank you for sharing!
\nI wonder how did you covert fp32 into int8?
@heoynwoo I used the same code shared above (you should set QUANTIZE = \"int8\").
\nTo cralify, int8 quantization result in about 20% slower inference time.
\nlocal evaluation time is:
\nint8 quantization turned out to be seriously slower in the Kaggle's evaluation environment.
\nit took far more longer time compared to local evaluation.
below is a comparison of the actual submission time:
\n| quantization | \nlocal eval time (sec) | \nestimated submission time (min) | \nactual submission time (min) | \n
|---|---|---|---|
| tf.float32 | \n167 | \n- | \n44 | \n
| tf.int8 | \n257 | \n67.7 | \n93 (timeout errror) | \n
Thank you! how did you measure actual submission time??
", "votes": 0, "replies": [] }, { "id": 2218894, "author_name": "Bilzard", "author_url": "", "post_date": "2023-04-12T05:47:48.570000", "content": "@heoynwoo You can poll the status of submission using Kaggle API.
\nVarious public notebooks exists:
\ne.g. https://www.kaggle.com/code/yasufuminakama/fb3-submission-time?scriptVersionId=104649325
In my experience, it all depends on the model. There are models that compress well and work well. Quantization can help in some cases, plus this is only the basic version, there are more complex options.
", "votes": 2, "replies": [] }, { "id": 2235463, "author_name": "coderRKJ", "author_url": "", "post_date": "2023-04-26T04:23:33.440000", "content": "In my specific case, submitting without any quantization results in normal result in about 45-60 mins while default (int8 quantization) optimization gave an output error in my model. Screenshot below:
\n