Those who are using Keras might be okay with this, but tensorflow users find it hard. By loading 2 embedding models, it consumes a lot of memory. Then, we need to create the training embedding out of it, then preprocess the text etc. \nThe main problem, is CUdNNgru / lstm is not reliable. For a tensorflow, bidirectional lstm, it takes more than 8000 seconds, which leads to time out. Because, it is not using CUdNNgru/lstm internally. Because, CUdNNgru/lstm has lot of issues like randomness, not able to deal with varying length sequences etc. So, thinking about ensembling is beyond the scope. Am i forced to use Keras?
", "messages": [ { "id": 431863, "postDate": "2018-12-03T03:09:05.793Z", "content": "Those who are using Keras might be okay with this, but tensorflow users find it hard. By loading 2 embedding models, it consumes a lot of memory. Then, we need to create the training embedding out of it, then preprocess the text etc. \nThe main problem, is CUdNNgru / lstm is not reliable. For a tensorflow, bidirectional lstm, it takes more than 8000 seconds, which leads to time out. Because, it is not using CUdNNgru/lstm internally. Because, CUdNNgru/lstm has lot of issues like randomness, not able to deal with varying length sequences etc. So, thinking about ensembling is beyond the scope. Am i forced to use Keras?
", "rawMarkdown": "Those who are using Keras might be okay with this, but tensorflow users find it hard. By loading 2 embedding models, it consumes a lot of memory. Then, we need to create the training embedding out of it, then preprocess the text etc. \nThe main problem, is CUdNNgru / lstm is not reliable. For a tensorflow, bidirectional lstm, it takes more than 8000 seconds, which leads to time out. Because, it is not using CUdNNgru/lstm internally. Because, CUdNNgru/lstm has lot of issues like randomness, not able to deal with varying length sequences etc. So, thinking about ensembling is beyond the scope. Am i forced to use Keras?", "votes": 3 }, { "id": 433352, "postDate": "2018-12-05T01:52:04.257Z", "content": "There shouldn't be any difference between Keras and TF since Keras is just a high-level wrapper for TF. I don't think there is a difference of speed between TF and Keras in terms of embedding and pre-processing.
\n\nTF also has CudnnLSTM/CudnnGRU. Tensorflow's native implementation of LSTM IS actually stable and reproducible. CuDnnLSTM is the one that is not reproducible.
", "rawMarkdown": "There shouldn't be any difference between Keras and TF since Keras is just a high-level wrapper for TF. I don't think there is a difference of speed between TF and Keras in terms of embedding and pre-processing.\n\nTF also has [CudnnLSTM/CudnnGRU](https://www.tensorflow.org/api_docs/python/tf/contrib/cudnn_rnn/CudnnLSTM). Tensorflow's [native implementation](https://www.tensorflow.org/api_docs/python/tf/nn/rnn_cell/LSTMCell) of LSTM IS actually stable and reproducible. CuDnnLSTM is the one that is not reproducible.", "votes": 1, "replies": [ { "id": 434448, "postDate": "2018-12-06T12:19:42.483Z", "content": "All keras code here uses CuDNNLSTM. Tensorflow have this, but it wont support variable size sentences.
", "rawMarkdown": "All keras code here uses CuDNNLSTM. Tensorflow have this, but it wont support variable size sentences." } ] }, { "id": 432086, "postDate": "2018-12-03T11:41:23.047Z", "content": "Could CudNNLSTM in Keras deal with varying length sequences?
", "rawMarkdown": "Could CudNNLSTM in Keras deal with varying length sequences? ", "replies": [ { "id": 432212, "postDate": "2018-12-03T15:03:06.303Z", "content": "AFAIK it should be fine if you just set the input shape to None for the sequence length
it won't help. AFAIK, it won't deal with varying length sequences
", "rawMarkdown": "it won't help. AFAIK, it won't deal with varying length sequences" }, { "id": 432408, "postDate": "2018-12-03T20:19:51.490Z", "content": "Sure it works, but you can't mask zeros.
", "rawMarkdown": "Sure it works, but you can't mask zeros." }, { "id": 432561, "postDate": "2018-12-04T03:31:48.933Z", "content": "How? In the tensorflow API, I have not seen a parameter to provide, sentence lengths explicitly.
\n\n\n__init__(\n num_layers,\n num_units,\n input_mode=CUDNN_INPUT_LINEAR_MODE,\n direction=CUDNN_RNN_UNIDIRECTION,\n dropout=0.0,\n seed=None,\n dtype=tf.float32,\n kernel_initializer=None,\n bias_initializer=None,\n name=None\n)\n
There shouldn't be any difference between Keras and TF since Keras is just a high-level wrapper for TF. I don't think there is a difference of speed between TF and Keras in terms of embedding and pre-processing.
\n\nTF also has CudnnLSTM/CudnnGRU. Tensorflow's native implementation of LSTM IS actually stable and reproducible. CuDnnLSTM is the one that is not reproducible.
", "votes": 1, "replies": [ { "id": 434448, "author_name": "aintnosunshine", "author_url": "", "post_date": "2018-12-06T12:19:42.483000", "content": "All keras code here uses CuDNNLSTM. Tensorflow have this, but it wont support variable size sentences.
", "votes": 0, "replies": [] } ] }, { "id": 432086, "author_name": "Lancelot0", "author_url": "", "post_date": "2018-12-03T11:41:23.047000", "content": "Could CudNNLSTM in Keras deal with varying length sequences?
", "votes": 0, "replies": [ { "id": 432212, "author_name": "Max Schumacher", "author_url": "", "post_date": "2018-12-03T15:03:06.303000", "content": "AFAIK it should be fine if you just set the input shape to None for the sequence length
it won't help. AFAIK, it won't deal with varying length sequences
", "votes": 0, "replies": [] }, { "id": 432408, "author_name": "Psi", "author_url": "", "post_date": "2018-12-03T20:19:51.490000", "content": "Sure it works, but you can't mask zeros.
", "votes": 0, "replies": [] }, { "id": 432561, "author_name": "aintnosunshine", "author_url": "", "post_date": "2018-12-04T03:31:48.933000", "content": "How? In the tensorflow API, I have not seen a parameter to provide, sentence lengths explicitly.
\n\n\n__init__(\n num_layers,\n num_units,\n input_mode=CUDNN_INPUT_LINEAR_MODE,\n direction=CUDNN_RNN_UNIDIRECTION,\n dropout=0.0,\n seed=None,\n dtype=tf.float32,\n kernel_initializer=None,\n bias_initializer=None,\n name=None\n)\n