{
  "id": 6846,
  "title": "Final Evaluation Phase",
  "url": "/competitions/flight2-main/discussion/6846",
  "author_name": "",
  "post_date": "2014-01-16T00:46:58.470Z",
  "votes": 1,
  "comment_count": 20,
  "views": 10968,
  "content": "<p>UPDATED:</p>\n<p>The Main phase of Flight Quest 2 closes this Saturday, January 18. We will then have a break while the data for the final evaluation period is collected. Here is a brief run down of what the Final Evaluation phase will look like.</p>\n<p>A new set of competition pages will be used for the Final phase (just as we did for the Milestone and the Main phases). Please re-accept the Rules and re-create the teams you had for FQ2-Main Phase.</p>\n<p>The final data set will be for data from January 18 to Feb 1 (<strong>15 days</strong>). We will provide one day of data for the training set and 14 days of data for the test set. As usual, the leaderboard will be based on the test set, and split into 1 day for public, 13 days for private.</p>\n<p>We will provide:</p>\n<ul>\n<li>One Day Simulator Files for the training day, this includes airports.csv, TestFlights.csv, etc.</li>\n<li>Full weather GRIB2 files for the training day.</li>\n<li>T=0 weather GRIB2 files for the test days.</li>\n<li>Flight Stats data for each day. Training day will have the full day. Test days will have data up to each day's cutoff time. Both data sets will include flights with either endpoint in the US.</li>\n<li>New turbulence zones for each day. (Restricted zones are unchanged.)</li>\n</ul>",
  "messages": [
    {
      "id": "37434",
      "postDate": "01/16/2014 00:46:58",
      "content": "<p>UPDATED:</p>\n<p>The Main phase of Flight Quest 2 closes this Saturday, January 18. We will then have a break while the data for the final evaluation period is collected. Here is a brief run down of what the Final Evaluation phase will look like.</p>\n<p>A new set of competition pages will be used for the Final phase (just as we did for the Milestone and the Main phases). Please re-accept the Rules and re-create the teams you had for FQ2-Main Phase.</p>\n<p>The final data set will be for data from January 18 to Feb 1 (<strong>15 days</strong>). We will provide one day of data for the training set and 14 days of data for the test set. As usual, the leaderboard will be based on the test set, and split into 1 day for public, 13 days for private.</p>\n<p>We will provide:</p>\n<ul>\n<li>One Day Simulator Files for the training day, this includes airports.csv, TestFlights.csv, etc.</li>\n<li>Full weather GRIB2 files for the training day.</li>\n<li>T=0 weather GRIB2 files for the test days.</li>\n<li>Flight Stats data for each day. Training day will have the full day. Test days will have data up to each day's cutoff time. Both data sets will include flights with either endpoint in the US.</li>\n<li>New turbulence zones for each day. (Restricted zones are unchanged.)</li>\n</ul>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37441",
      "postDate": "01/16/2014 06:33:46",
      "content": "<p>What is the point of the public/private split?</p>\n<p>Correct me if I'm wrong, but I assume the models will be fixed (we're not allowed to change them), so there should be at most 1 submission for each team.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37444",
      "postDate": "01/16/2014 07:47:42",
      "content": "<p>Unless the 13 private days become public once the winners are announced, nobody will be able to check the real classification... I'm with charango, what's the point of having 2 leaderboards when no changes to the model are allowed?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37446",
      "postDate": "01/16/2014 08:19:16",
      "content": "<p>Like charango and Gabriele Seppi, I don't understand the reason for the split, unless the original plan to simply run the final model submitted during the current (Main) phase is being revised.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37447",
      "postDate": "01/16/2014 08:19:28",
      "content": "<p>Regardless of the public/private split, I would request that you provide data for 1 training day and 14 test days (just like what was done for the current leaderboard). That way we can run our current models without making any code changes.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37465",
      "postDate": "01/16/2014 15:47:45",
      "content": "<p>I assume that the public part of the final leaderboard is intended as a &quot;sanity check&quot; and an antidote to some of the heartburn encountered in Flight quest 1. IIRC, in that contest, some of the best models got scuttled by trivial things like confusion over which time zone to use.</p>\n<p>If half of your planes crash on the final leaderboard, you probably need to fix a &quot;bug&quot; and then be prepared to plead your case to Kaggle (if you're in the money). I can think of legitimate reasons why one might want to make a couple of preliminary submissions just to rule out problems early.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37468",
      "postDate": "01/16/2014 19:07:30",
      "content": "<p>I see a airports.csv will be provided for the public day. I think a lot of people (perhaps implicitly) depend on the airports file to remain the same, especially the radius and height columns for safe landing. Is that a valid assumption or should we prepare for changes? (Which will be almost impossible to test properly.)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37471",
      "postDate": "01/16/2014 19:36:16",
      "content": "<p>[quote=Willem Mestrom;37468]</p>\n<p>I see a airports.csv will be provided for the public day. I think a lot of people (perhaps implicitly) depend on the airports file to remain the same, especially the radius and height columns for safe landing. Is that a valid assumption or should we prepare for changes?&nbsp;</p>\n<p>[/quote]</p>\n<p>Airports.csv, Landing.csv, Taxi.csv, Projection.json, restrictedZones.csv, and Configuration.json will not change.</p>\n<p>[quote=Anil Thomas;37447]</p>\n<p>Regardless of the public/private split, I would request that you provide data for 1 training day and 14 test days (just like what was done for the current leaderboard). That way we can run our current models without making any code changes.</p>\n<p>[/quote]</p>\n<p>See updated opening post.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37473",
      "postDate": "01/16/2014 20:25:52",
      "content": "<p>[quote=&#8469;ick &#8712; Sardo;37471]</p>\n<p>Just to reiterate this point to everyone: the <strong>&quot;One Day Simulator Files&quot; will be the first day of the 14 test days</strong>. &nbsp;The provided groundConditions, actualLandings, actualTakeoffs, and weather files will be the same ones used by the evaluation simulator. &nbsp;You're responsible for predicting the conditions (should your agent need them) for the remaining&nbsp;<strong>13 days. &nbsp;&nbsp;</strong>Please make any necessary code changes before the model deadline. </p>\n<p>[/quote]</p>\n<p>What are we supposed to do with the first day? Should it be part of the final submission? Should we make two separate submissions? Is it okay to ignore the simulator files and make normal predictions for all 14 days?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37475",
      "postDate": "01/16/2014 20:32:05",
      "content": "<p>Make one submission for all 14 days as you normally do for leaderboards. We're just saying that the split is 1/13 (versus 7/7 as previously used) and that you'll get One Day Simulator files for that 1 public day.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37478",
      "postDate": "01/16/2014 21:18:54",
      "content": "<p>[quote=joycenv;37475]</p>\n<p>Make one submission for all 14 days as you normally do for leaderboards. We're just saying that the split is 1/13 (versus 7/7 as previously used) and that you'll get One Day Simulator files for that 1 public day.</p>\n<p>[/quote]</p>\n<p>Just to be clear: is it permissible for our models to read the One Day Simulator files, in order to use the public day as an additional &quot;training day&quot; that could help improve performance on the other 13 days? (This could be very useful since the public day would be much closer temporally to the private days than any of the 3 training sets released prior to the code freeze, which are months old.)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37479",
      "postDate": "01/16/2014 21:38:16",
      "content": "<p>I should expand on my question a bit more. I'm considering creating a setting in my configuration file called &quot;training_day_cutoff&quot; (or something like that). For the purpose of generating a submission to the current leaderboard, this could be set to &quot;2013-09-10 18:03:53&quot; (since we already have One Day Simulator files for September 10). Then, for generating a submission to the final evaluation leaderboard, this setting could be changed to &quot;2014-01-19 XX:XX:XX&quot;. That way, the model could train on January 19 using my exact same code that has already been written to train on September 10.</p>\n<p>Would I be allowed to do that?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37481",
      "postDate": "01/16/2014 23:25:03",
      "content": "<p>I have updated the original post that started this thread. Please review that post for the most current information about the Final Evaluation Phase of the competition.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37489",
      "postDate": "01/17/2014 07:42:38",
      "content": "<p>I still don't get the usefulness of the split leaderboard. Doesn't it add complexity to an otherwise pretty straightforward phase?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37497",
      "postDate": "01/17/2014 09:44:10",
      "content": "<p>what about Weather Forecast GRIB2 Files for&nbsp; the final data set</p>\n<p>can we use them ? are you going to provide them ?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37503",
      "postDate": "01/17/2014 10:43:26",
      "content": "<p>Hi joycenv,</p>\n<p>Why do we have to accept the rules again, are there any changes?<br> &nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37505",
      "postDate": "01/17/2014 11:42:08",
      "content": "<p>Never a dull moment with this contest! :)</p>\n<p><br>Last night I went to bed thinking that today, I would have to hack my test&nbsp;script to allow overriding predictions with Kaggle-provided historical data. Today, I'm happy I didn't rush to do it, since it no longer seems to be necessary. Thanks, but...</p>\n<p><br>... I am still confused: what is the training day data&nbsp;for?</p>\n<p><br>As far as I can see, our models must be frozen by the submission deadline tomorrow. In particular, any GRIB2 files used in training must be archived and a hash of the archive included as part of the submission in &quot;DataHash.txt&quot;, according to the instructions at</p>\n<p>http://www.gequest.com/c/flight2-main/details/submission-instructions</p>\n<p>So we won't be able to use the training data for&nbsp;January 18 to update our models. What for, then?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37516",
      "postDate": "01/17/2014 16:52:48",
      "content": "<p>[quote=Jules;37503]</p>\n<p>Hi joycenv,</p>\n<p>Why do we have to accept the rules again, are there any changes?<br> &nbsp;</p>\n<p>[/quote]</p>\n<p>Hi Jules,</p>\n<p>There are no changes. Having to accept the Rules again is because of a limitation with our platform -- I cannot easily migrate everyone over to a new leaderboard in a clean way so I am creating a new competition that is a continuation of this competition. This is good in some ways (for comp administration), but annoying for others, like requesting that participants re-accept (the same) Rules and recreate (the same) teams. Sorry for the overhead on your part, I admit it's a little bit of a hack.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37519",
      "postDate": "01/17/2014 18:10:41",
      "content": "<p>[quote=TecS;37505]</p>\n<p><span style=\"line-height: 1.4\">As far as I can see, our models must be frozen by the submission deadline tomorrow. In particular, any GRIB2 files used in training must be archived and a hash of the archive included as part of the submission in &quot;DataHash.txt&quot;, according to the instructions at</span></p>\n<p>So we won't be able to use the training data for&nbsp;January 18 to update our models. What for, then?</p>\n<p>[/quote]</p>\n<p>You can use the upcoming weather data for training; you just don't need to hash it.</p>\n<p>https://www.gequest.com/c/flight2-main/forums/t/6764/final-model-agent-submissions/37117#post37117</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37532",
      "postDate": "01/17/2014 19:52:49",
      "content": "<p>Question about &quot;README file with instructions on how to do so&quot;.</p>\n<p>It's allowed to do simple manual actions like - fill file with new cutoff times(I think all need something like this - because of it's basic settings), copy files and make folders?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "38307",
      "postDate": "02/03/2014 15:59:28",
      "content": "<p>[quote=joycenv;37434]</p>\n<p>Test days will have data up to each day's cutoff time. </p>\n<p>[/quote]</p>\n<p>When you'll select cutoff times please keep in mind that weather grib data for 2014.01.26 not exists for 19 hour. http://soostrc.comet.ucar.edu/data/grib/rap/20140126/hybrid/</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 37441,
      "author_name": "sergeykozub",
      "author_url": "",
      "post_date": "01/16/2014 06:33:46",
      "content": "<p>What is the point of the public/private split?</p>\n<p>Correct me if I'm wrong, but I assume the models will be fixed (we're not allowed to change them), so there should be at most 1 submission for each team.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37444,
      "author_name": "gabrieleseppi",
      "author_url": "",
      "post_date": "01/16/2014 07:47:42",
      "content": "<p>Unless the 13 private days become public once the winners are announced, nobody will be able to check the real classification... I'm with charango, what's the point of having 2 leaderboards when no changes to the model are allowed?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37446,
      "author_name": "hyperborean",
      "author_url": "",
      "post_date": "01/16/2014 08:19:16",
      "content": "<p>Like charango and Gabriele Seppi, I don't understand the reason for the split, unless the original plan to simply run the final model submitted during the current (Main) phase is being revised.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37447,
      "author_name": "anlthms",
      "author_url": "",
      "post_date": "01/16/2014 08:19:28",
      "content": "<p>Regardless of the public/private split, I would request that you provide data for 1 training day and 14 test days (just like what was done for the current leaderboard). That way we can run our current models without making any code changes.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37465,
      "author_name": "glazed",
      "author_url": "",
      "post_date": "01/16/2014 15:47:45",
      "content": "<p>I assume that the public part of the final leaderboard is intended as a &quot;sanity check&quot; and an antidote to some of the heartburn encountered in Flight quest 1. IIRC, in that contest, some of the best models got scuttled by trivial things like confusion over which time zone to use.</p>\n<p>If half of your planes crash on the final leaderboard, you probably need to fix a &quot;bug&quot; and then be prepared to plead your case to Kaggle (if you're in the money). I can think of legitimate reasons why one might want to make a couple of preliminary submissions just to rule out problems early.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37468,
      "author_name": "wmestrom",
      "author_url": "",
      "post_date": "01/16/2014 19:07:30",
      "content": "<p>I see a airports.csv will be provided for the public day. I think a lot of people (perhaps implicitly) depend on the airports file to remain the same, especially the radius and height columns for safe landing. Is that a valid assumption or should we prepare for changes? (Which will be almost impossible to test properly.)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37471,
      "author_name": "nicksardo",
      "author_url": "",
      "post_date": "01/16/2014 19:36:16",
      "content": "<p>[quote=Willem Mestrom;37468]</p>\n<p>I see a airports.csv will be provided for the public day. I think a lot of people (perhaps implicitly) depend on the airports file to remain the same, especially the radius and height columns for safe landing. Is that a valid assumption or should we prepare for changes?&nbsp;</p>\n<p>[/quote]</p>\n<p>Airports.csv, Landing.csv, Taxi.csv, Projection.json, restrictedZones.csv, and Configuration.json will not change.</p>\n<p>[quote=Anil Thomas;37447]</p>\n<p>Regardless of the public/private split, I would request that you provide data for 1 training day and 14 test days (just like what was done for the current leaderboard). That way we can run our current models without making any code changes.</p>\n<p>[/quote]</p>\n<p>See updated opening post.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37473,
      "author_name": "wmestrom",
      "author_url": "",
      "post_date": "01/16/2014 20:25:52",
      "content": "<p>[quote=&#8469;ick &#8712; Sardo;37471]</p>\n<p>Just to reiterate this point to everyone: the <strong>&quot;One Day Simulator Files&quot; will be the first day of the 14 test days</strong>. &nbsp;The provided groundConditions, actualLandings, actualTakeoffs, and weather files will be the same ones used by the evaluation simulator. &nbsp;You're responsible for predicting the conditions (should your agent need them) for the remaining&nbsp;<strong>13 days. &nbsp;&nbsp;</strong>Please make any necessary code changes before the model deadline. </p>\n<p>[/quote]</p>\n<p>What are we supposed to do with the first day? Should it be part of the final submission? Should we make two separate submissions? Is it okay to ignore the simulator files and make normal predictions for all 14 days?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37475,
      "author_name": "joycenv",
      "author_url": "",
      "post_date": "01/16/2014 20:32:05",
      "content": "<p>Make one submission for all 14 days as you normally do for leaderboards. We're just saying that the split is 1/13 (versus 7/7 as previously used) and that you'll get One Day Simulator files for that 1 public day.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37478,
      "author_name": "dougkoch",
      "author_url": "",
      "post_date": "01/16/2014 21:18:54",
      "content": "<p>[quote=joycenv;37475]</p>\n<p>Make one submission for all 14 days as you normally do for leaderboards. We're just saying that the split is 1/13 (versus 7/7 as previously used) and that you'll get One Day Simulator files for that 1 public day.</p>\n<p>[/quote]</p>\n<p>Just to be clear: is it permissible for our models to read the One Day Simulator files, in order to use the public day as an additional &quot;training day&quot; that could help improve performance on the other 13 days? (This could be very useful since the public day would be much closer temporally to the private days than any of the 3 training sets released prior to the code freeze, which are months old.)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37479,
      "author_name": "dougkoch",
      "author_url": "",
      "post_date": "01/16/2014 21:38:16",
      "content": "<p>I should expand on my question a bit more. I'm considering creating a setting in my configuration file called &quot;training_day_cutoff&quot; (or something like that). For the purpose of generating a submission to the current leaderboard, this could be set to &quot;2013-09-10 18:03:53&quot; (since we already have One Day Simulator files for September 10). Then, for generating a submission to the final evaluation leaderboard, this setting could be changed to &quot;2014-01-19 XX:XX:XX&quot;. That way, the model could train on January 19 using my exact same code that has already been written to train on September 10.</p>\n<p>Would I be allowed to do that?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37481,
      "author_name": "joycenv",
      "author_url": "",
      "post_date": "01/16/2014 23:25:03",
      "content": "<p>I have updated the original post that started this thread. Please review that post for the most current information about the Final Evaluation Phase of the competition.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37489,
      "author_name": "gabrieleseppi",
      "author_url": "",
      "post_date": "01/17/2014 07:42:38",
      "content": "<p>I still don't get the usefulness of the split leaderboard. Doesn't it add complexity to an otherwise pretty straightforward phase?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37497,
      "author_name": "drorrr",
      "author_url": "",
      "post_date": "01/17/2014 09:44:10",
      "content": "<p>what about Weather Forecast GRIB2 Files for&nbsp; the final data set</p>\n<p>can we use them ? are you going to provide them ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37503,
      "author_name": "julesvanligtenberg",
      "author_url": "",
      "post_date": "01/17/2014 10:43:26",
      "content": "<p>Hi joycenv,</p>\n<p>Why do we have to accept the rules again, are there any changes?<br> &nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37505,
      "author_name": "hyperborean",
      "author_url": "",
      "post_date": "01/17/2014 11:42:08",
      "content": "<p>Never a dull moment with this contest! :)</p>\n<p><br>Last night I went to bed thinking that today, I would have to hack my test&nbsp;script to allow overriding predictions with Kaggle-provided historical data. Today, I'm happy I didn't rush to do it, since it no longer seems to be necessary. Thanks, but...</p>\n<p><br>... I am still confused: what is the training day data&nbsp;for?</p>\n<p><br>As far as I can see, our models must be frozen by the submission deadline tomorrow. In particular, any GRIB2 files used in training must be archived and a hash of the archive included as part of the submission in &quot;DataHash.txt&quot;, according to the instructions at</p>\n<p>http://www.gequest.com/c/flight2-main/details/submission-instructions</p>\n<p>So we won't be able to use the training data for&nbsp;January 18 to update our models. What for, then?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37516,
      "author_name": "joycenv",
      "author_url": "",
      "post_date": "01/17/2014 16:52:48",
      "content": "<p>[quote=Jules;37503]</p>\n<p>Hi joycenv,</p>\n<p>Why do we have to accept the rules again, are there any changes?<br> &nbsp;</p>\n<p>[/quote]</p>\n<p>Hi Jules,</p>\n<p>There are no changes. Having to accept the Rules again is because of a limitation with our platform -- I cannot easily migrate everyone over to a new leaderboard in a clean way so I am creating a new competition that is a continuation of this competition. This is good in some ways (for comp administration), but annoying for others, like requesting that participants re-accept (the same) Rules and recreate (the same) teams. Sorry for the overhead on your part, I admit it's a little bit of a hack.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37519,
      "author_name": "nicksardo",
      "author_url": "",
      "post_date": "01/17/2014 18:10:41",
      "content": "<p>[quote=TecS;37505]</p>\n<p><span style=\"line-height: 1.4\">As far as I can see, our models must be frozen by the submission deadline tomorrow. In particular, any GRIB2 files used in training must be archived and a hash of the archive included as part of the submission in &quot;DataHash.txt&quot;, according to the instructions at</span></p>\n<p>So we won't be able to use the training data for&nbsp;January 18 to update our models. What for, then?</p>\n<p>[/quote]</p>\n<p>You can use the upcoming weather data for training; you just don't need to hash it.</p>\n<p>https://www.gequest.com/c/flight2-main/forums/t/6764/final-model-agent-submissions/37117#post37117</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37532,
      "author_name": "dmytrolystopad",
      "author_url": "",
      "post_date": "01/17/2014 19:52:49",
      "content": "<p>Question about &quot;README file with instructions on how to do so&quot;.</p>\n<p>It's allowed to do simple manual actions like - fill file with new cutoff times(I think all need something like this - because of it's basic settings), copy files and make folders?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 38307,
      "author_name": "dmytrolystopad",
      "author_url": "",
      "post_date": "02/03/2014 15:59:28",
      "content": "<p>[quote=joycenv;37434]</p>\n<p>Test days will have data up to each day's cutoff time. </p>\n<p>[/quote]</p>\n<p>When you'll select cutoff times please keep in mind that weather grib data for 2014.01.26 not exists for 19 hour. http://soostrc.comet.ucar.edu/data/grib/rap/20140126/hybrid/</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "37434": "",
    "37441": "",
    "37444": "",
    "37446": "",
    "37447": "",
    "37465": "",
    "37468": "",
    "37471": "",
    "37473": "",
    "37475": "",
    "37478": "",
    "37479": "",
    "37481": "",
    "37489": "",
    "37497": "",
    "37503": "",
    "37505": "",
    "37516": "",
    "37519": "",
    "37532": "",
    "38307": ""
  },
  "source": "meta"
}