Is it troublesome to use? Or other reasons?
", "messages": [ { "id": "588301", "postDate": "07/30/2019 11:53:43", "content": "Is it troublesome to use? Or other reasons?
", "rawMarkdown": "Is it troublesome to use? Or other reasons?", "votes": null }, { "id": "588303", "postDate": "07/30/2019 12:00:03", "content": "I want to complete this game without using keras and slim, but how to submit a prediction using a trained model is really a problem. Can someone give a suggestion?\n```\nN = epcoh*LEARNING_RATE_STEP
\n\ninit = tf.global_variables_initializer()\ny = tf.nn.softmax(out)\nwith tf.Session() as sess:\n coord = tf.train.Coordinator() # 设置多线程协调器\n threads = tf.train.start_queue_runners(sess=sess, coord=coord)\n init.run()\n #saver = tf.train.Saver()\n #tf.add_to_collection('pred_network', y)\n #summary_merge = tf.summary.merge_all()\n #f_summary = tf.summary.FileWriter(logs_train_dir, graph=sess.graph)\n for i in range(N):
\n\n datas, labels = sess.run(next_element)\n _, tra_loss, acc = sess.run(\n [train_step, cross_entropy, accuracy], feed_dict={xs: datas, ys: labels, keep_prob: 0.5,is_training:True})\n #summary_tmp = sess.run(summary_merge, feed_dict={\n #xs: datas, ys: labels, keep_prob: 0.5,is_training:True}) # 计算summary\n learning_val = sess.run(learning_rate)\n #f_summary.add_summary(summary=summary_tmp,\n #global_step=i/400) # 写入summary\n learning_rate = sess.run([learning_rate])\n if i % 1 == 0:\n print(\"After %s steps: acc is %f, learning rage is %f, loss is %f\" % (\n i, acc,learning_val ,tra_loss))\n #checkpoint_path = os.path.join(logs_train_dir, 'thing.ckpt')\n #saver.save(sess, checkpoint_path)\ncoord.join(threads)\nsess.close()\n```
", "rawMarkdown": "I want to complete this game without using keras and slim, but how to submit a prediction using a trained model is really a problem. Can someone give a suggestion?\n```\nN = epcoh*LEARNING_RATE_STEP\n#logs_train_dir = '../input/'\ninit = tf.global_variables_initializer()\ny = tf.nn.softmax(out)\nwith tf.Session() as sess:\n coord = tf.train.Coordinator() # 设置多线程协调器\n threads = tf.train.start_queue_runners(sess=sess, coord=coord)\n init.run()\n #saver = tf.train.Saver()\n #tf.add_to_collection('pred_network', y)\n #summary_merge = tf.summary.merge_all()\n #f_summary = tf.summary.FileWriter(logs_train_dir, graph=sess.graph)\n for i in range(N):\n \n datas, labels = sess.run(next_element)\n _, tra_loss, acc = sess.run(\n [train_step, cross_entropy, accuracy], feed_dict={xs: datas, ys: labels, keep_prob: 0.5,is_training:True})\n #summary_tmp = sess.run(summary_merge, feed_dict={\n #xs: datas, ys: labels, keep_prob: 0.5,is_training:True}) # 计算summary\n learning_val = sess.run(learning_rate)\n #f_summary.add_summary(summary=summary_tmp,\n #global_step=i/400) # 写入summary\n learning_rate = sess.run([learning_rate])\n if i % 1 == 0:\n print(\"After %s steps: acc is %f, learning rage is %f, loss is %f\" % (\n i, acc,learning_val ,tra_loss))\n #checkpoint_path = os.path.join(logs_train_dir, 'thing.ckpt')\n #saver.save(sess, checkpoint_path)\n coord.join(threads)\n sess.close()\n```", "votes": null }, { "id": "588888", "postDate": "07/31/2019 07:43:51", "content": "Load your trained model as data set, then add it to your kernel.
", "rawMarkdown": "Load your trained model as data set, then add it to your kernel.", "votes": null }, { "id": "588906", "postDate": "07/31/2019 08:05:02", "content": "For me, it's just because keras is simple and good looking.\nBut sometimes, there are lot of custom layers, or some tensorflow ops that need to embedded in graph while deploying to prevent training-serving skew(may be some pre/post processing), in those case I freeze keras model in protobuf and use it to do inference.\nIn this competition, you can train model on your local machine or separate kaggle kernel, then write a kernel for only inference part. For this, you can freeze tensorflow graph, upload it as dataset and use it to infer. \nI wrote below script for batch prediction using multiple models. May be you can modify it for your need:\n```\ndef load_graph(graph_path):\n with tf.gfile.GFile(graph_path, \"rb\") as f:\n graph_def = tf.GraphDef()\n graph_def.ParseFromString(f.read())
\n\nwith tf.Graph().as_default() as graph:\n tf.import_graph_def(graph_def, name=f\"\")\nreturn graph\nprint('\\n++++++++++++++++++++++ After Freezing ++++++++++++++++++++++')\n input_tensors = ['input_bpe:0', 'clf_input_extra:0']\n output_tensors = ['clf_output/Sigmoid:0']\n print(\n f'\\ninput node names: {input_tensors}\\noutput node names: {output_tensors}')
\n\nnum_splits = math.ceil(len(val_data.index) / batch_size)\nval_splits = np.array_split(val_data, num_splits)\n\nfeed_dict = {}\nK.clear_session()\ngraph_paths = [\n 'models/freezed/model0.pb',\n 'models/freezed/model1.pb',\n 'models/freezed/model2.pb',\n 'models/freezed/model3.pb',\n]\nfor i, path in enumerate(tqdm(graph_paths)):\n tfgraph = load_graph(path)\n table_init_op = tfgraph.get_operation_by_name('init_all_tables')\n with tf.Session(graph=tfgraph) as session:\n session.run(table_init_op)\n preds = []\n for split in tqdm(val_splits):\n x = split['comment_text'].values[:, np.newaxis]\n x_e = split[['qm_ratio', 'em_ratio']].values\n feed_dict[input_tensors[0]] = x\n feed_dict[input_tensors[1]] = x_e\n pred = session.run(output_tensors, feed_dict=feed_dict)[0]\n preds += pred.tolist()\n preds = np.asarray(preds)[:, 0]\n result = evaluator.get_final_metric(preds)\n print(f'result of model {i}: {result}')\n predictions[:, i] = preds\npredictions = np.mean(predictions, axis=-1)\nresult = evaluator.get_final_metric(predictions)\nprint(f'\\nResult after averaging predictions: {result}\\n')\n```
", "rawMarkdown": "For me, it's just because keras is simple and good looking.\nBut sometimes, there are lot of custom layers, or some tensorflow ops that need to embedded in graph while deploying to prevent training-serving skew(may be some pre/post processing), in those case I freeze keras model in protobuf and use it to do inference.\nIn this competition, you can train model on your local machine or separate kaggle kernel, then write a kernel for only inference part. For this, you can freeze tensorflow graph, upload it as dataset and use it to infer. \nI wrote below script for batch prediction using multiple models. May be you can modify it for your need:\n```\ndef load_graph(graph_path):\n with tf.gfile.GFile(graph_path, \"rb\") as f:\n graph_def = tf.GraphDef()\n graph_def.ParseFromString(f.read())\n\n with tf.Graph().as_default() as graph:\n tf.import_graph_def(graph_def, name=f\"\")\n return graph\n\nprint('\\n++++++++++++++++++++++ After Freezing ++++++++++++++++++++++')\n input_tensors = ['input_bpe:0', 'clf_input_extra:0']\n output_tensors = ['clf_output/Sigmoid:0']\n print(\n f'\\ninput node names: {input_tensors}\\noutput node names: {output_tensors}')\n\n num_splits = math.ceil(len(val_data.index) / batch_size)\n val_splits = np.array_split(val_data, num_splits)\n\n feed_dict = {}\n K.clear_session()\n graph_paths = [\n 'models/freezed/model0.pb',\n 'models/freezed/model1.pb',\n 'models/freezed/model2.pb',\n 'models/freezed/model3.pb',\n ]\n for i, path in enumerate(tqdm(graph_paths)):\n tfgraph = load_graph(path)\n table_init_op = tfgraph.get_operation_by_name('init_all_tables')\n with tf.Session(graph=tfgraph) as session:\n session.run(table_init_op)\n preds = []\n for split in tqdm(val_splits):\n x = split['comment_text'].values[:, np.newaxis]\n x_e = split[['qm_ratio', 'em_ratio']].values\n feed_dict[input_tensors[0]] = x\n feed_dict[input_tensors[1]] = x_e\n pred = session.run(output_tensors, feed_dict=feed_dict)[0]\n preds += pred.tolist()\n preds = np.asarray(preds)[:, 0]\n result = evaluator.get_final_metric(preds)\n print(f'result of model {i}: {result}')\n predictions[:, i] = preds\n predictions = np.mean(predictions, axis=-1)\n result = evaluator.get_final_metric(predictions)\n print(f'\\nResult after averaging predictions: {result}\\n')\n```", "votes": null }, { "id": "588963", "postDate": "07/31/2019 09:25:37", "content": "At the moment it seems that there is only this way.
", "rawMarkdown": "At the moment it seems that there is only this way.", "votes": null }, { "id": "588967", "postDate": "07/31/2019 09:31:04", "content": "Thanks for your suggestion, I need more complicated image preprocessing, so I made a piece of code that makes TF.DATA.DATSTSET support CV2 operations.\n\ndef _read_py_function(filename, label):\n train_img=[]\n radius = 1
\n n_points = 8 * radius \n img = cv2.imread(filename.decode(),cv2.IMREAD_UNCHANGED)\n img = cv2.resize(img,(224,224))\n img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n img = local_binary_pattern(img, n_points, radius) \n img = img.reshape(224,224,1)/255\n #img = np.concatenate((img, img, img), axis=-1)\n img=np.array(img,dtype=\"uint8\")\n return img, label\ndef _read_image_caller(filename, label):\n return tuple(tf.py_func(_read_py_function, [filename, label], [tf.uint8, label.dtype]))\n
This is not freezable, so you can do this preprocessing in inference kernel and feed resulting numpy array to tf graph using feed_dict.\nSorry, If I misunderstood your problem.
", "rawMarkdown": "This is not freezable, so you can do this preprocessing in inference kernel and feed resulting numpy array to tf graph using feed_dict.\nSorry, If I misunderstood your problem.", "votes": null }, { "id": "589047", "postDate": "07/31/2019 11:39:02", "content": "You are right, thank you very much for helping me a lot!
", "rawMarkdown": "You are right, thank you very much for helping me a lot!", "votes": null } ], "comments": [ { "id": 588303, "author_name": "a18974761777", "author_url": "", "post_date": "07/30/2019 12:00:03", "content": "I want to complete this game without using keras and slim, but how to submit a prediction using a trained model is really a problem. Can someone give a suggestion?\n```\nN = epcoh*LEARNING_RATE_STEP
\n\ninit = tf.global_variables_initializer()\ny = tf.nn.softmax(out)\nwith tf.Session() as sess:\n coord = tf.train.Coordinator() # 设置多线程协调器\n threads = tf.train.start_queue_runners(sess=sess, coord=coord)\n init.run()\n #saver = tf.train.Saver()\n #tf.add_to_collection('pred_network', y)\n #summary_merge = tf.summary.merge_all()\n #f_summary = tf.summary.FileWriter(logs_train_dir, graph=sess.graph)\n for i in range(N):
\n\n datas, labels = sess.run(next_element)\n _, tra_loss, acc = sess.run(\n [train_step, cross_entropy, accuracy], feed_dict={xs: datas, ys: labels, keep_prob: 0.5,is_training:True})\n #summary_tmp = sess.run(summary_merge, feed_dict={\n #xs: datas, ys: labels, keep_prob: 0.5,is_training:True}) # 计算summary\n learning_val = sess.run(learning_rate)\n #f_summary.add_summary(summary=summary_tmp,\n #global_step=i/400) # 写入summary\n learning_rate = sess.run([learning_rate])\n if i % 1 == 0:\n print(\"After %s steps: acc is %f, learning rage is %f, loss is %f\" % (\n i, acc,learning_val ,tra_loss))\n #checkpoint_path = os.path.join(logs_train_dir, 'thing.ckpt')\n #saver.save(sess, checkpoint_path)\ncoord.join(threads)\nsess.close()\n```
", "votes": null, "replies": [ { "id": 588888, "author_name": "surfhigh", "author_url": "", "post_date": "07/31/2019 07:43:51", "content": "Load your trained model as data set, then add it to your kernel.
", "votes": null, "replies": [] }, { "id": 588963, "author_name": "a18974761777", "author_url": "", "post_date": "07/31/2019 09:25:37", "content": "At the moment it seems that there is only this way.
", "votes": null, "replies": [] } ] }, { "id": 588906, "author_name": "ashwan1", "author_url": "", "post_date": "07/31/2019 08:05:02", "content": "For me, it's just because keras is simple and good looking.\nBut sometimes, there are lot of custom layers, or some tensorflow ops that need to embedded in graph while deploying to prevent training-serving skew(may be some pre/post processing), in those case I freeze keras model in protobuf and use it to do inference.\nIn this competition, you can train model on your local machine or separate kaggle kernel, then write a kernel for only inference part. For this, you can freeze tensorflow graph, upload it as dataset and use it to infer. \nI wrote below script for batch prediction using multiple models. May be you can modify it for your need:\n```\ndef load_graph(graph_path):\n with tf.gfile.GFile(graph_path, \"rb\") as f:\n graph_def = tf.GraphDef()\n graph_def.ParseFromString(f.read())
\n\nwith tf.Graph().as_default() as graph:\n tf.import_graph_def(graph_def, name=f\"\")\nreturn graph\nprint('\\n++++++++++++++++++++++ After Freezing ++++++++++++++++++++++')\n input_tensors = ['input_bpe:0', 'clf_input_extra:0']\n output_tensors = ['clf_output/Sigmoid:0']\n print(\n f'\\ninput node names: {input_tensors}\\noutput node names: {output_tensors}')
\n\nnum_splits = math.ceil(len(val_data.index) / batch_size)\nval_splits = np.array_split(val_data, num_splits)\n\nfeed_dict = {}\nK.clear_session()\ngraph_paths = [\n 'models/freezed/model0.pb',\n 'models/freezed/model1.pb',\n 'models/freezed/model2.pb',\n 'models/freezed/model3.pb',\n]\nfor i, path in enumerate(tqdm(graph_paths)):\n tfgraph = load_graph(path)\n table_init_op = tfgraph.get_operation_by_name('init_all_tables')\n with tf.Session(graph=tfgraph) as session:\n session.run(table_init_op)\n preds = []\n for split in tqdm(val_splits):\n x = split['comment_text'].values[:, np.newaxis]\n x_e = split[['qm_ratio', 'em_ratio']].values\n feed_dict[input_tensors[0]] = x\n feed_dict[input_tensors[1]] = x_e\n pred = session.run(output_tensors, feed_dict=feed_dict)[0]\n preds += pred.tolist()\n preds = np.asarray(preds)[:, 0]\n result = evaluator.get_final_metric(preds)\n print(f'result of model {i}: {result}')\n predictions[:, i] = preds\npredictions = np.mean(predictions, axis=-1)\nresult = evaluator.get_final_metric(predictions)\nprint(f'\\nResult after averaging predictions: {result}\\n')\n```
", "votes": null, "replies": [ { "id": 588967, "author_name": "a18974761777", "author_url": "", "post_date": "07/31/2019 09:31:04", "content": "Thanks for your suggestion, I need more complicated image preprocessing, so I made a piece of code that makes TF.DATA.DATSTSET support CV2 operations.\n\ndef _read_py_function(filename, label):\n train_img=[]\n radius = 1
\n n_points = 8 * radius \n img = cv2.imread(filename.decode(),cv2.IMREAD_UNCHANGED)\n img = cv2.resize(img,(224,224))\n img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n img = local_binary_pattern(img, n_points, radius) \n img = img.reshape(224,224,1)/255\n #img = np.concatenate((img, img, img), axis=-1)\n img=np.array(img,dtype=\"uint8\")\n return img, label\ndef _read_image_caller(filename, label):\n return tuple(tf.py_func(_read_py_function, [filename, label], [tf.uint8, label.dtype]))\n
This is not freezable, so you can do this preprocessing in inference kernel and feed resulting numpy array to tf graph using feed_dict.\nSorry, If I misunderstood your problem.
", "votes": null, "replies": [] }, { "id": 589047, "author_name": "a18974761777", "author_url": "", "post_date": "07/31/2019 11:39:02", "content": "You are right, thank you very much for helping me a lot!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "588301": "Is it troublesome to use? Or other reasons?", "588303": "I want to complete this game without using keras and slim, but how to submit a prediction using a trained model is really a problem. Can someone give a suggestion?\n```\nN = epcoh*LEARNING_RATE_STEP\n#logs_train_dir = '../input/'\ninit = tf.global_variables_initializer()\ny = tf.nn.softmax(out)\nwith tf.Session() as sess:\n coord = tf.train.Coordinator() # 设置多线程协调器\n threads = tf.train.start_queue_runners(sess=sess, coord=coord)\n init.run()\n #saver = tf.train.Saver()\n #tf.add_to_collection('pred_network', y)\n #summary_merge = tf.summary.merge_all()\n #f_summary = tf.summary.FileWriter(logs_train_dir, graph=sess.graph)\n for i in range(N):\n \n datas, labels = sess.run(next_element)\n _, tra_loss, acc = sess.run(\n [train_step, cross_entropy, accuracy], feed_dict={xs: datas, ys: labels, keep_prob: 0.5,is_training:True})\n #summary_tmp = sess.run(summary_merge, feed_dict={\n #xs: datas, ys: labels, keep_prob: 0.5,is_training:True}) # 计算summary\n learning_val = sess.run(learning_rate)\n #f_summary.add_summary(summary=summary_tmp,\n #global_step=i/400) # 写入summary\n learning_rate = sess.run([learning_rate])\n if i % 1 == 0:\n print(\"After %s steps: acc is %f, learning rage is %f, loss is %f\" % (\n i, acc,learning_val ,tra_loss))\n #checkpoint_path = os.path.join(logs_train_dir, 'thing.ckpt')\n #saver.save(sess, checkpoint_path)\n coord.join(threads)\n sess.close()\n```", "588888": "Load your trained model as data set, then add it to your kernel.", "588906": "For me, it's just because keras is simple and good looking.\nBut sometimes, there are lot of custom layers, or some tensorflow ops that need to embedded in graph while deploying to prevent training-serving skew(may be some pre/post processing), in those case I freeze keras model in protobuf and use it to do inference.\nIn this competition, you can train model on your local machine or separate kaggle kernel, then write a kernel for only inference part. For this, you can freeze tensorflow graph, upload it as dataset and use it to infer. \nI wrote below script for batch prediction using multiple models. May be you can modify it for your need:\n```\ndef load_graph(graph_path):\n with tf.gfile.GFile(graph_path, \"rb\") as f:\n graph_def = tf.GraphDef()\n graph_def.ParseFromString(f.read())\n\n with tf.Graph().as_default() as graph:\n tf.import_graph_def(graph_def, name=f\"\")\n return graph\n\nprint('\\n++++++++++++++++++++++ After Freezing ++++++++++++++++++++++')\n input_tensors = ['input_bpe:0', 'clf_input_extra:0']\n output_tensors = ['clf_output/Sigmoid:0']\n print(\n f'\\ninput node names: {input_tensors}\\noutput node names: {output_tensors}')\n\n num_splits = math.ceil(len(val_data.index) / batch_size)\n val_splits = np.array_split(val_data, num_splits)\n\n feed_dict = {}\n K.clear_session()\n graph_paths = [\n 'models/freezed/model0.pb',\n 'models/freezed/model1.pb',\n 'models/freezed/model2.pb',\n 'models/freezed/model3.pb',\n ]\n for i, path in enumerate(tqdm(graph_paths)):\n tfgraph = load_graph(path)\n table_init_op = tfgraph.get_operation_by_name('init_all_tables')\n with tf.Session(graph=tfgraph) as session:\n session.run(table_init_op)\n preds = []\n for split in tqdm(val_splits):\n x = split['comment_text'].values[:, np.newaxis]\n x_e = split[['qm_ratio', 'em_ratio']].values\n feed_dict[input_tensors[0]] = x\n feed_dict[input_tensors[1]] = x_e\n pred = session.run(output_tensors, feed_dict=feed_dict)[0]\n preds += pred.tolist()\n preds = np.asarray(preds)[:, 0]\n result = evaluator.get_final_metric(preds)\n print(f'result of model {i}: {result}')\n predictions[:, i] = preds\n predictions = np.mean(predictions, axis=-1)\n result = evaluator.get_final_metric(predictions)\n print(f'\\nResult after averaging predictions: {result}\\n')\n```", "588963": "At the moment it seems that there is only this way.", "588967": "Thanks for your suggestion, I need more complicated image preprocessing, so I made a piece of code that makes TF.DATA.DATSTSET support CV2 operations.\n```\ndef _read_py_function(filename, label):\n train_img=[]\n radius = 1 \n n_points = 8 * radius \n img = cv2.imread(filename.decode(),cv2.IMREAD_UNCHANGED)\n img = cv2.resize(img,(224,224))\n img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n img = local_binary_pattern(img, n_points, radius) \n img = img.reshape(224,224,1)/255\n #img = np.concatenate((img, img, img), axis=-1)\n img=np.array(img,dtype=\"uint8\")\n return img, label\ndef _read_image_caller(filename, label):\n return tuple(tf.py_func(_read_py_function, [filename, label], [tf.uint8, label.dtype]))\n```", "589001": "This is not freezable, so you can do this preprocessing in inference kernel and feed resulting numpy array to tf graph using feed_dict.\nSorry, If I misunderstood your problem.", "589047": "You are right, thank you very much for helping me a lot!" }, "source": "meta" }