{ "id": 60742, "title": "Fine-tune VGG16 in Keras + MLFlow LB0.62", "url": "/competitions/state-farm-distracted-driver-detection/discussion/60742", "author_name": "Cyril P", "post_date": "2018-07-09T10:06:11.064000", "votes": 2, "comment_count": 15, "views": 0, "content": "

Hello,

\n\n

I wanted to share my very simple pipeline for this project.

\n\n

I used a VGG16 in Keras. I fine tuned only the last 3 conv layers and the last 3 fully connected layers. Using only 9 epochs and a very small learning rate (0.00005). No cross-validation. No splitting by driver.

\n\n

I also used the new MLFlow tool to define a pipeline: training, predicting and submitting. \nOne can configure the parameters in the file config.py and then simply run the pipeline using mlflow run.

\n\n

Here is my code:

\n\n\n\n

or

\n\n", "messages": [ { "id": 354318, "postDate": "2018-07-09T10:06:11.063Z", "content": "

Hello,

\n\n

I wanted to share my very simple pipeline for this project.

\n\n

I used a VGG16 in Keras. I fine tuned only the last 3 conv layers and the last 3 fully connected layers. Using only 9 epochs and a very small learning rate (0.00005). No cross-validation. No splitting by driver.

\n\n

I also used the new MLFlow tool to define a pipeline: training, predicting and submitting. \nOne can configure the parameters in the file config.py and then simply run the pipeline using mlflow run.

\n\n

Here is my code:

\n\n\n\n

or

\n\n", "rawMarkdown": "Hello,\n\nI wanted to share my very simple pipeline for this project.\n\nI used a VGG16 in Keras. I fine tuned only the last 3 conv layers and the last 3 fully connected layers. Using only 9 epochs and a very small learning rate (0.00005). No cross-validation. No splitting by driver.\n\n\n\nI also used the new [MLFlow][1] tool to define a pipeline: training, predicting and submitting. \nOne can configure the parameters in the file config.py and then simply run the pipeline using `mlflow run`.\n\nHere is my code:\n\n - [Demonstration in a Jupyter notebook][2] (not the code used for the submission, just take it as an example)\n\nor\n\n - [`train.py` script][3]\n - [`predict.py` script][4]\n\n\n [1]: https://mlflow.org/\n [2]: http://nbviewer.jupyter.org/github/cyril-p/DistractedDriverDetection/blob/master/main_training.ipynb\n [3]: https://github.com/cyril-p/DistractedDriverDetection/blob/master/src/train.py\n [4]: https://github.com/cyril-p/DistractedDriverDetection/blob/master/src/predict.py", "votes": 2 }, { "id": 424912, "postDate": "2018-11-20T21:48:00.927Z", "content": "

Hi, it seems there is no label for test data. Could you please tell me how to calculate the loss and accuracy of the model?

", "rawMarkdown": "Hi, it seems there is no label for test data. Could you please tell me how to calculate the loss and accuracy of the model?" }, { "id": 404942, "postDate": "2018-10-16T16:06:13.117Z", "content": "

Hi, Cyril. If it is OK, can you share the file \"Model/VGG16_lr0.0001_train3_epochs4_data_aug0.h5\"?

", "rawMarkdown": "Hi, Cyril. If it is OK, can you share the file \"Model/VGG16_lr0.0001_train3_epochs4_data_aug0.h5\"? " }, { "id": 404698, "postDate": "2018-10-16T08:25:27.927Z", "content": "

Hi,
\nCan I use google cloud platform free for this big dataset? It seems that the size of the dataset has exceeded the range of the GCP's free support.

", "rawMarkdown": "Hi, \nCan I use google cloud platform free for this big dataset? It seems that the size of the dataset has exceeded the range of the GCP's free support." }, { "id": 397611, "postDate": "2018-10-02T19:19:52.497Z", "content": "

Here is the error:\nTraceback (most recent call last):\n File \"./train.py\", line 214, in \n use_multiprocessing=True)\n File \"/usr/local/lib/python3.5/dist-packages/keras/legacy/interfaces.py\", line 91, in wrapper\n return func(*args, **kwargs)\n File \"/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\", line 1415, in fit_generator\n initial_epoch=initial_epoch)\n File \"/usr/local/lib/python3.5/dist-packages/keras/engine/training_generator.py\", line 213, in fit_generator\n class_weight=class_weight)\n File \"/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\", line 1215, in train_on_batch\n outputs = self.train_function(ins)\n File \"/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py\", line 2666, in call\n return self._call(inputs)\n File \"/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py\", line 2636, in _call\n fetched = self._callable_fn(*array_vals)\n File \"/home/gpu_user/tensorflow18/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1454, in call\n self._session._session, self._handle, args, status, None)\n File \"/home/gpu_user/tensorflow18/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py\", line 519, in exit\n c_api.TF_GetCode(self.status.status))\ntensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[48,64,224,224] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc\n [[Node: block1_conv2/convolution = Conv2D[T=DT_FLOAT, data_format=\"NCHW\", dilations=[1, 1, 1, 1], padding=\"SAME\", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](block1_conv1/Relu, block1_conv2/kernel/read)]]\nHint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

\n\n
 [[Node: loss/mul/_283 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device_incarnation=1, tensor_name=\"edge_648_loss/mul\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]()]]\n
\n\n

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

", "rawMarkdown": "Here is the error:\nTraceback (most recent call last):\n File \"./train.py\", line 214, in ", "replies": [ { "id": 397613, "postDate": "2018-10-02T19:25:23.827Z", "content": "

You are running out of memory and using a CPU for the training. I used a big GPU on a Google Cloud Platform for this project. howverr, you could decrease n_layers_train (the number of layers to be trained), this should help.

", "rawMarkdown": "You are running out of memory and using a CPU for the training. I used a big GPU on a Google Cloud Platform for this project. howverr, you could decrease n_layers_train (the number of layers to be trained), this should help." } ] }, { "id": 397606, "postDate": "2018-10-02T19:03:02.220Z", "content": "

Hi, I try it on a big machine but it can just run 1/4 epoch and then aborted. The batch_size is set to 48. And the size of the dataset is 1000.

", "rawMarkdown": "Hi, I try it on a big machine but it can just run 1/4 epoch and then aborted. The batch_size is set to 48. And the size of the dataset is 1000." }, { "id": 396847, "postDate": "2018-10-01T13:30:56.593Z", "content": "

Hi! I meet this problem when running the train.py. Can you please help me?\nUsing TensorFlow backend.\nLoading dataset train...\n('folder train/c0', 'loaded')\n('folder train/c1', 'loaded')\n('folder train/c2', 'loaded')\n('folder train/c3', 'loaded')\n('folder train/c4', 'loaded')\n('folder train/c5', 'loaded')\n('folder train/c6', 'loaded')\n('folder train/c7', 'loaded')\n('folder train/c8', 'loaded')\n('folder train/c9', 'loaded')\nTraceback (most recent call last):\n File \"./train.py\", line 155, in \n X_train_, y_train_ = load_train_dataset(dataset_dir_train, img_reshape_size) \n File \"./train.py\", line 93, in load_train_dataset\n X = np.asarray(X, dtype=np.float16)\n File \"/usr/local/lib/python2.7/dist-packages/numpy/core/numeric.py\", line 501, in asarray\n return array(a, dtype, copy=False, order=order)\nMemoryError

", "rawMarkdown": "Hi! I meet this problem when running the train.py. Can you please help me?\nUsing TensorFlow backend.\nLoading dataset train...\n('folder train/c0', 'loaded')\n('folder train/c1', 'loaded')\n('folder train/c2', 'loaded')\n('folder train/c3', 'loaded')\n('folder train/c4', 'loaded')\n('folder train/c5', 'loaded')\n('folder train/c6', 'loaded')\n('folder train/c7', 'loaded')\n('folder train/c8', 'loaded')\n('folder train/c9', 'loaded')\nTraceback (most recent call last):\n File \"./train.py\", line 155, in ", "replies": [ { "id": 396871, "postDate": "2018-10-01T14:03:28.787Z", "content": "

Hi Jamie, this means that you don't have enough memory (RAM I guess) to load the dataset. Try on a bigger machine or adapt my code with the examples presented elsewhere on this forum to load the data by batch. Thank you !

", "rawMarkdown": "Hi Jamie, this means that you don't have enough memory (RAM I guess) to load the dataset. Try on a bigger machine or adapt my code with the examples presented elsewhere on this forum to load the data by batch. Thank you !", "votes": 1 }, { "id": 396873, "postDate": "2018-10-01T14:06:42.647Z", "content": "

Thank you.

", "rawMarkdown": "Thank you. " } ] }, { "id": 365689, "postDate": "2018-08-03T07:35:49.460Z", "content": "

Hi, Thank you for your sharing. It can be seen in your jupyter notebook that about 99% classification accuracy was achieved by your model. I didn't go deeper. Why your result only ranks 30% since so high accuracy is achieved.\nThank you!

", "rawMarkdown": "Hi, Thank you for your sharing. It can be seen in your jupyter notebook that about 99% classification accuracy was achieved by your model. I didn't go deeper. Why your result only ranks 30% since so high accuracy is achieved.\nThank you!", "replies": [ { "id": 365938, "postDate": "2018-08-03T17:34:41.547Z", "content": "

Hi, thank you for your comment ! Here the log-loss is taken as the evaluation metric. I invite you to read how it is computed but the fact is that you could have a very high accuracy and a non-zero log loss. The accuracy uses the the class predicted (obtained using the argmax of the predicted probabilities). The log-loss uses the probability. for instance:

\n\n\n\n

The accuracy would be 1, but the log-loss would be around 0.3

", "rawMarkdown": "Hi, thank you for your comment ! Here the log-loss is taken as the evaluation metric. I invite you to read how it is computed but the fact is that you could have a very high accuracy and a non-zero log loss. The accuracy uses the the **class predicted** (obtained using the argmax of the predicted probabilities). The log-loss uses the **probability**. for instance:\n\n - Correct class: Cat\n - Predicted probabilites:\n - cat: 0.7 | dog:0.3 \n - Predicted class (using the argmax): Cat\n\nThe accuracy would be 1, but the log-loss would be around 0.3", "votes": 1 }, { "id": 366090, "postDate": "2018-08-04T02:06:31.760Z", "content": "

Thank you for your applying. In that case, can we think that the value loss is not reasonable? In my opinion, classification accuracy is more important than vague log loss. (Forgive me for my bad English.)

", "rawMarkdown": "Thank you for your applying. In that case, can we think that the value loss is not reasonable? In my opinion, classification accuracy is more important than vague log loss. (Forgive me for my bad English.)" } ] }, { "id": 397662, "postDate": "2018-10-02T21:57:26.133Z", "rawMarkdown": "", "isDeleted": true }, { "id": 396821, "postDate": "2018-10-01T12:43:34.827Z", "rawMarkdown": "", "isDeleted": true }, { "id": 397864, "postDate": "2018-10-03T08:40:15.890Z", "content": "

Run them successfully, Thank you.

", "rawMarkdown": "Run them successfully, Thank you." } ], "comments": [ { "id": 424912, "author_name": "jiegenghua", "author_url": "", "post_date": "2018-11-20T21:48:00.927000", "content": "

Hi, it seems there is no label for test data. Could you please tell me how to calculate the loss and accuracy of the model?

", "votes": 0, "replies": [] }, { "id": 404942, "author_name": "", "author_url": "", "post_date": "2018-10-16T16:06:13.117000", "content": "

Hi, Cyril. If it is OK, can you share the file \"Model/VGG16_lr0.0001_train3_epochs4_data_aug0.h5\"?

", "votes": 0, "replies": [] }, { "id": 404698, "author_name": "", "author_url": "", "post_date": "2018-10-16T08:25:27.927000", "content": "

Hi,
\nCan I use google cloud platform free for this big dataset? It seems that the size of the dataset has exceeded the range of the GCP's free support.

", "votes": 0, "replies": [] }, { "id": 397611, "author_name": "", "author_url": "", "post_date": "2018-10-02T19:19:52.497000", "content": "

Here is the error:\nTraceback (most recent call last):\n File \"./train.py\", line 214, in \n use_multiprocessing=True)\n File \"/usr/local/lib/python3.5/dist-packages/keras/legacy/interfaces.py\", line 91, in wrapper\n return func(*args, **kwargs)\n File \"/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\", line 1415, in fit_generator\n initial_epoch=initial_epoch)\n File \"/usr/local/lib/python3.5/dist-packages/keras/engine/training_generator.py\", line 213, in fit_generator\n class_weight=class_weight)\n File \"/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\", line 1215, in train_on_batch\n outputs = self.train_function(ins)\n File \"/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py\", line 2666, in call\n return self._call(inputs)\n File \"/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py\", line 2636, in _call\n fetched = self._callable_fn(*array_vals)\n File \"/home/gpu_user/tensorflow18/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1454, in call\n self._session._session, self._handle, args, status, None)\n File \"/home/gpu_user/tensorflow18/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py\", line 519, in exit\n c_api.TF_GetCode(self.status.status))\ntensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[48,64,224,224] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc\n [[Node: block1_conv2/convolution = Conv2D[T=DT_FLOAT, data_format=\"NCHW\", dilations=[1, 1, 1, 1], padding=\"SAME\", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](block1_conv1/Relu, block1_conv2/kernel/read)]]\nHint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

\n\n
 [[Node: loss/mul/_283 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device_incarnation=1, tensor_name=\"edge_648_loss/mul\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]()]]\n
\n\n

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

", "votes": 0, "replies": [ { "id": 397613, "author_name": "Cyril P", "author_url": "", "post_date": "2018-10-02T19:25:23.827000", "content": "

You are running out of memory and using a CPU for the training. I used a big GPU on a Google Cloud Platform for this project. howverr, you could decrease n_layers_train (the number of layers to be trained), this should help.

", "votes": 0, "replies": [] } ] }, { "id": 397606, "author_name": "", "author_url": "", "post_date": "2018-10-02T19:03:02.220000", "content": "

Hi, I try it on a big machine but it can just run 1/4 epoch and then aborted. The batch_size is set to 48. And the size of the dataset is 1000.

", "votes": 0, "replies": [] }, { "id": 396847, "author_name": "", "author_url": "", "post_date": "2018-10-01T13:30:56.593000", "content": "

Hi! I meet this problem when running the train.py. Can you please help me?\nUsing TensorFlow backend.\nLoading dataset train...\n('folder train/c0', 'loaded')\n('folder train/c1', 'loaded')\n('folder train/c2', 'loaded')\n('folder train/c3', 'loaded')\n('folder train/c4', 'loaded')\n('folder train/c5', 'loaded')\n('folder train/c6', 'loaded')\n('folder train/c7', 'loaded')\n('folder train/c8', 'loaded')\n('folder train/c9', 'loaded')\nTraceback (most recent call last):\n File \"./train.py\", line 155, in \n X_train_, y_train_ = load_train_dataset(dataset_dir_train, img_reshape_size) \n File \"./train.py\", line 93, in load_train_dataset\n X = np.asarray(X, dtype=np.float16)\n File \"/usr/local/lib/python2.7/dist-packages/numpy/core/numeric.py\", line 501, in asarray\n return array(a, dtype, copy=False, order=order)\nMemoryError

", "votes": 0, "replies": [ { "id": 396871, "author_name": "Cyril P", "author_url": "", "post_date": "2018-10-01T14:03:28.787000", "content": "

Hi Jamie, this means that you don't have enough memory (RAM I guess) to load the dataset. Try on a bigger machine or adapt my code with the examples presented elsewhere on this forum to load the data by batch. Thank you !

", "votes": 1, "replies": [] }, { "id": 396873, "author_name": "", "author_url": "", "post_date": "2018-10-01T14:06:42.647000", "content": "

Thank you.

", "votes": 0, "replies": [] } ] }, { "id": 365689, "author_name": "Jia LI", "author_url": "", "post_date": "2018-08-03T07:35:49.460000", "content": "

Hi, Thank you for your sharing. It can be seen in your jupyter notebook that about 99% classification accuracy was achieved by your model. I didn't go deeper. Why your result only ranks 30% since so high accuracy is achieved.\nThank you!

", "votes": 0, "replies": [ { "id": 365938, "author_name": "Cyril P", "author_url": "", "post_date": "2018-08-03T17:34:41.547000", "content": "

Hi, thank you for your comment ! Here the log-loss is taken as the evaluation metric. I invite you to read how it is computed but the fact is that you could have a very high accuracy and a non-zero log loss. The accuracy uses the the class predicted (obtained using the argmax of the predicted probabilities). The log-loss uses the probability. for instance:

\n\n\n\n

The accuracy would be 1, but the log-loss would be around 0.3

", "votes": 1, "replies": [] }, { "id": 366090, "author_name": "Jia LI", "author_url": "", "post_date": "2018-08-04T02:06:31.760000", "content": "

Thank you for your applying. In that case, can we think that the value loss is not reasonable? In my opinion, classification accuracy is more important than vague log loss. (Forgive me for my bad English.)

", "votes": 0, "replies": [] } ] }, { "id": 397662, "author_name": "", "author_url": "", "post_date": "2018-10-02T21:57:26.133000", "content": "", "votes": 0, "replies": [] }, { "id": 396821, "author_name": "", "author_url": "", "post_date": "2018-10-01T12:43:34.827000", "content": "", "votes": 0, "replies": [] }, { "id": 397864, "author_name": "", "author_url": "", "post_date": "2018-10-03T08:40:15.890000", "content": "

Run them successfully, Thank you.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "354318": "Hello,\n\nI wanted to share my very simple pipeline for this project.\n\nI used a VGG16 in Keras. I fine tuned only the last 3 conv layers and the last 3 fully connected layers. Using only 9 epochs and a very small learning rate (0.00005). No cross-validation. No splitting by driver.\n\n\n\nI also used the new [MLFlow][1] tool to define a pipeline: training, predicting and submitting. \nOne can configure the parameters in the file config.py and then simply run the pipeline using `mlflow run`.\n\nHere is my code:\n\n - [Demonstration in a Jupyter notebook][2] (not the code used for the submission, just take it as an example)\n\nor\n\n - [`train.py` script][3]\n - [`predict.py` script][4]\n\n\n [1]: https://mlflow.org/\n [2]: http://nbviewer.jupyter.org/github/cyril-p/DistractedDriverDetection/blob/master/main_training.ipynb\n [3]: https://github.com/cyril-p/DistractedDriverDetection/blob/master/src/train.py\n [4]: https://github.com/cyril-p/DistractedDriverDetection/blob/master/src/predict.py", "424912": "Hi, it seems there is no label for test data. Could you please tell me how to calculate the loss and accuracy of the model?", "404942": "Hi, Cyril. If it is OK, can you share the file \"Model/VGG16_lr0.0001_train3_epochs4_data_aug0.h5\"? ", "404698": "Hi, \nCan I use google cloud platform free for this big dataset? It seems that the size of the dataset has exceeded the range of the GCP's free support.", "397611": "Here is the error:\nTraceback (most recent call last):\n File \"./train.py\", line 214, in ", "397606": "Hi, I try it on a big machine but it can just run 1/4 epoch and then aborted. The batch_size is set to 48. And the size of the dataset is 1000.", "396847": "Hi! I meet this problem when running the train.py. Can you please help me?\nUsing TensorFlow backend.\nLoading dataset train...\n('folder train/c0', 'loaded')\n('folder train/c1', 'loaded')\n('folder train/c2', 'loaded')\n('folder train/c3', 'loaded')\n('folder train/c4', 'loaded')\n('folder train/c5', 'loaded')\n('folder train/c6', 'loaded')\n('folder train/c7', 'loaded')\n('folder train/c8', 'loaded')\n('folder train/c9', 'loaded')\nTraceback (most recent call last):\n File \"./train.py\", line 155, in ", "365689": "Hi, Thank you for your sharing. It can be seen in your jupyter notebook that about 99% classification accuracy was achieved by your model. I didn't go deeper. Why your result only ranks 30% since so high accuracy is achieved.\nThank you!", "397662": "", "396821": "", "397864": "Run them successfully, Thank you." } }