{
  "id": 545856,
  "title": "Tip: How to Log Your Experiments and Results",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/545856",
  "author_name": "",
  "post_date": "2024-11-12T13:11:11.683338Z",
  "votes": 2,
  "comment_count": 1,
  "views": 0,
  "content": "<p>It's a big challenge for me to keep everything organized — there are a lot of experiments, parameters, data splits, etc.</p>\n<p>A great tool that helps me stay organized is MLflow, which many people in modeling and data science are probably already familiar with.</p>\n<p>MLflow has a framework called Tracking, which you can use to log almost everything related to your model: parameters, loss functions, and more. I am currently using it here for neural networks (NN):</p>\n<p>Example:</p>\n<pre><code>\n\n mlflow\n mlflow.keras\n\n\nlog_name =  + datetime.now().strftime()\nmlflow.set_experiment(log_name)\n\n\n\n (tf.keras.callbacks.Callback):\n     ():\n        mlflow.log_metrics({\n           : logs[],\n           : logs[],\n           : logs[],\n           : logs[],\n\n    }, step=epoch)\n\n mlflow.start_run():\n    \n        mlflow.log_param(, learning_rate)\n        mlflow.log_param(, batch_size)\n        mlflow.keras.log_model(model, )\n        mlflow.log_param(, optimizer.get_config())\n        mlflow.log_param(, loss_fn)\n        mlflow.log_param(, model.to_json())\n\n\n      \n\n      mlflow.log_metric(, r2_eval) \n</code></pre>\n<p>I'm not an expert on MLflow, but it helps a lot (and ChatGPT helped me build this code).</p>",
  "messages": [
    {
      "id": "3043519",
      "postDate": "11/12/2024 13:11:11",
      "content": "<p>It's a big challenge for me to keep everything organized — there are a lot of experiments, parameters, data splits, etc.</p>\n<p>A great tool that helps me stay organized is MLflow, which many people in modeling and data science are probably already familiar with.</p>\n<p>MLflow has a framework called Tracking, which you can use to log almost everything related to your model: parameters, loss functions, and more. I am currently using it here for neural networks (NN):</p>\n<p>Example:</p>\n<pre><code>\n\n mlflow\n mlflow.keras\n\n\nlog_name =  + datetime.now().strftime()\nmlflow.set_experiment(log_name)\n\n\n\n (tf.keras.callbacks.Callback):\n     ():\n        mlflow.log_metrics({\n           : logs[],\n           : logs[],\n           : logs[],\n           : logs[],\n\n    }, step=epoch)\n\n mlflow.start_run():\n    \n        mlflow.log_param(, learning_rate)\n        mlflow.log_param(, batch_size)\n        mlflow.keras.log_model(model, )\n        mlflow.log_param(, optimizer.get_config())\n        mlflow.log_param(, loss_fn)\n        mlflow.log_param(, model.to_json())\n\n\n      \n\n      mlflow.log_metric(, r2_eval) \n</code></pre>\n<p>I'm not an expert on MLflow, but it helps a lot (and ChatGPT helped me build this code).</p>",
      "rawMarkdown": "It's a big challenge for me to keep everything organized — there are a lot of experiments, parameters, data splits, etc.\n\nA great tool that helps me stay organized is MLflow, which many people in modeling and data science are probably already familiar with.\n\nMLflow has a framework called Tracking, which you can use to log almost everything related to your model: parameters, loss functions, and more. I am currently using it here for neural networks (NN):\n\nExample:\n\n\n```python\n#!pip install mlflow\n\nimport mlflow\nimport mlflow.keras\n\n\nlog_name = \"log_model\" + datetime.now().strftime(\"%Y_%m_%d_%H_%M\")\nmlflow.set_experiment(log_name)\n\n\n#Create a customcallback to log the data and see the train curves\nclass CustomCallback(tf.keras.callbacks.Callback):\n    def on_epoch_end(self, epoch, logs=None):\n        mlflow.log_metrics({\n           \"loss\": logs[\"loss\"],\n           \"r2\": logs[\"action_r2\"],\n           \"val_loss\": logs[\"val_loss\"],\n           \"val_r2\": logs[\"val_action_r2\"],\n\n    }, step=epoch)\n\nwith mlflow.start_run():\n    # Registrar parâmetros\n        mlflow.log_param(\"learning_rate\", learning_rate)\n        mlflow.log_param(\"batch_size\", batch_size)\n        mlflow.keras.log_model(model, \"modelo_keras\")\n        mlflow.log_param(\"optimizer\", optimizer.get_config())\n        mlflow.log_param(\"loss_function\", loss_fn)\n        mlflow.log_param(\"model_architecture\", model.to_json())\n\n\n      #your model\n\n      mlflow.log_metric('oot_r2', r2_eval) \n```\n\n\n\n\nI'm not an expert on MLflow, but it helps a lot (and ChatGPT helped me build this code).",
      "votes": null
    },
    {
      "id": "3043953",
      "postDate": "11/12/2024 20:40:21",
      "content": "<p>Cool! I had been using Neptune.ai for tracking my experiments and it's quite nice for me. Definitely will take a look at MLflow. </p>",
      "rawMarkdown": "Cool! I had been using Neptune.ai for tracking my experiments and it's quite nice for me. Definitely will take a look at MLflow.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3043953,
      "author_name": "yeraypabongonzalez",
      "author_url": "",
      "post_date": "11/12/2024 20:40:21",
      "content": "<p>Cool! I had been using Neptune.ai for tracking my experiments and it's quite nice for me. Definitely will take a look at MLflow. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3043519": "It's a big challenge for me to keep everything organized — there are a lot of experiments, parameters, data splits, etc.\n\nA great tool that helps me stay organized is MLflow, which many people in modeling and data science are probably already familiar with.\n\nMLflow has a framework called Tracking, which you can use to log almost everything related to your model: parameters, loss functions, and more. I am currently using it here for neural networks (NN):\n\nExample:\n\n\n```python\n#!pip install mlflow\n\nimport mlflow\nimport mlflow.keras\n\n\nlog_name = \"log_model\" + datetime.now().strftime(\"%Y_%m_%d_%H_%M\")\nmlflow.set_experiment(log_name)\n\n\n#Create a customcallback to log the data and see the train curves\nclass CustomCallback(tf.keras.callbacks.Callback):\n    def on_epoch_end(self, epoch, logs=None):\n        mlflow.log_metrics({\n           \"loss\": logs[\"loss\"],\n           \"r2\": logs[\"action_r2\"],\n           \"val_loss\": logs[\"val_loss\"],\n           \"val_r2\": logs[\"val_action_r2\"],\n\n    }, step=epoch)\n\nwith mlflow.start_run():\n    # Registrar parâmetros\n        mlflow.log_param(\"learning_rate\", learning_rate)\n        mlflow.log_param(\"batch_size\", batch_size)\n        mlflow.keras.log_model(model, \"modelo_keras\")\n        mlflow.log_param(\"optimizer\", optimizer.get_config())\n        mlflow.log_param(\"loss_function\", loss_fn)\n        mlflow.log_param(\"model_architecture\", model.to_json())\n\n\n      #your model\n\n      mlflow.log_metric('oot_r2', r2_eval) \n```\n\n\n\n\nI'm not an expert on MLflow, but it helps a lot (and ChatGPT helped me build this code).",
    "3043953": "Cool! I had been using Neptune.ai for tracking my experiments and it's quite nice for me. Definitely will take a look at MLflow."
  },
  "source": "meta"
}