{
  "id": 2987,
  "title": "Private Leaderboard Data Release",
  "url": "/competitions/predict-closed-questions-on-stack-overflow/discussion/2987",
  "author_name": "",
  "post_date": "2012-10-25T09:07:56.747Z",
  "votes": null,
  "comment_count": 21,
  "views": 8940,
  "content": "<p>Hi all,<br>\r\n<br>\r\nI have just uploaded the private leaderboard data, which contains the questions asked between October 10 and October 23.<br>\r\n<br>\r\nApologies for the short delay - I had intended to release the data on October 24 PT, but my preliminary testing with the new set uncovered a pesky bug in the final submission process. Final submissions currently remain disabled, and they should be enabled by\r\n the end of the day on October 25, once the fix goes live. To ensure that you have a full week to submit your final predictions, I have shifted the deadline to make your final submission to November 3.<br>\r\n<br>\r\nThe new training data (up through October 9) will be released ASAP, along with an updated stratified sample. You may optionally retrain your model using the updated training set. However, you should not be manually updating or changing the model in any other\r\n way.<br>\r\n<br>\r\nIn order to receive a final score and rank for this competition, you should re-run your model on the private leaderboard set &amp; make your final submission.<br>\r\n<br>\r\nAlso, the visualization contest is closing in less than 2 days! Submit any interesting visualizations or insights you have found on this problem. The ones submitted so far have just touched the surface of the richness in this dataset.</p>\r\n<p>Cheers,</p>\r\n<p>Ben</p>",
  "messages": [
    {
      "id": "16194",
      "postDate": "10/25/2012 09:07:56",
      "content": "<p>Hi all,<br>\r\n<br>\r\nI have just uploaded the private leaderboard data, which contains the questions asked between October 10 and October 23.<br>\r\n<br>\r\nApologies for the short delay - I had intended to release the data on October 24 PT, but my preliminary testing with the new set uncovered a pesky bug in the final submission process. Final submissions currently remain disabled, and they should be enabled by\r\n the end of the day on October 25, once the fix goes live. To ensure that you have a full week to submit your final predictions, I have shifted the deadline to make your final submission to November 3.<br>\r\n<br>\r\nThe new training data (up through October 9) will be released ASAP, along with an updated stratified sample. You may optionally retrain your model using the updated training set. However, you should not be manually updating or changing the model in any other\r\n way.<br>\r\n<br>\r\nIn order to receive a final score and rank for this competition, you should re-run your model on the private leaderboard set &amp; make your final submission.<br>\r\n<br>\r\nAlso, the visualization contest is closing in less than 2 days! Submit any interesting visualizations or insights you have found on this problem. The ones submitted so far have just touched the surface of the richness in this dataset.</p>\r\n<p>Cheers,</p>\r\n<p>Ben</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16202",
      "postDate": "10/25/2012 14:13:34",
      "content": "<p>Call me picky, but I'd welcome you using a more standard notation for dates, instead of 10/9 or 11/3 - not all contestants are American. In fact, when you look at the TOP 30 users, only a third are. And half of them have names that don't sound native at\r\n all.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16220",
      "postDate": "10/26/2012 02:13:18",
      "content": "<p>Hi Ben</p>\r\n<p>Does &quot;<span>Raw TrainingSet</span>&quot; contain the new training data? I find that the &quot;CloseReason&quot; in many questions are empty. Could you have a look? Thanks! &nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16226",
      "postDate": "10/26/2012 05:09:14",
      "content": "<p>Yes, this new data set has a different schema than before. Instead of OpenStatus you have IsClosed and CloseReason. Do we need to update our parsers for the training set, then?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16227",
      "postDate": "10/26/2012 05:22:17",
      "content": "<p>[quote=Foxtrot;16202]</p>\r\n<p>Call me picky, but I'd welcome you using a more standard notation for dates, instead of 10/9 or 11/3 - not all contestants are American. In fact, when you look at the TOP 30 users, only a third are. And half of them have names that don't sound native at\r\n all.</p>\r\n<p>[/quote]Thanks for pointing this out - I have updated my post accordingly and will remember this for the future. Have asked my team to do the same.</p>\r\n<p>I lived in Switzerland for a year &amp; know it's confusing seeing 11/3 and wondering if it means November 3 or March 11.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16228",
      "postDate": "10/26/2012 05:27:51",
      "content": "<p>[quote=Erheng Zhong;16220]</p>\r\n<p>Hi Ben</p>\r\n<p>Does &quot;<span>Raw TrainingSet</span>&quot; contain the new training data? I find that the &quot;CloseReason&quot; in many questions are empty. Could you have a look? Thanks! &nbsp;</p>\r\n<p>[/quote]That was the format StackOverflow had originally provided the data in. I'd modified it to make it slightly easier to work with (so the ground truth was in a single column). The reformatted file is up as train_October_9_2012.csv. I'm currently travelling\r\n and on a flaky hotel internet connection, but other compressed formats of the same file will be appearing in the next 6 hours.</p>\r\n<p>Additionally, I've recreated the stratified sample from the new training set (train-sample_October_9_2012.csv) in case anyone's finals models used the subsampled set.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16229",
      "postDate": "10/26/2012 05:29:03",
      "content": "<p>[quote=Andy Sloane;16226]</p>\r\n<p>Yes, this new data set has a different schema than before. Instead of OpenStatus you have IsClosed and CloseReason. Do we need to update our parsers for the training set, then?</p>\r\n<p>[/quote]No, use train_October_9_2012.csv (just went live on the data page)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16231",
      "postDate": "10/26/2012 07:03:21",
      "content": "<p>Can we make multiple submissions and select one later in this phase too? By the way, there leaderboard will not be updated until the submissions are closed, right?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16232",
      "postDate": "10/26/2012 07:43:33",
      "content": "<p>Replying to myself, my original train of thought was that I may decide to submit the results from the model that was trained on the data up to august, if I see that the distribution of classes is more similar that way. But since that goes against the spirit\r\n of the competition, I'll submit the results of the version retrained on the oct9 dataset.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16235",
      "postDate": "10/26/2012 08:52:49",
      "content": "<p>[quote=Ben Hamner;16228]</p>\r\n<p>Additionally, I've recreated the stratified sample from the new training set (train-sample_October_9_2012.csv) in case anyone's finals models used the subsampled set.</p>\r\n<p>[/quote]</p>\r\n<p>train-sample_October_9_2012.csv apparently has a lot of &quot;nan&quot;s where the original train-sample.csv had empty CSV fields. So it's still either a preprocessing script or modifications to our parsers, it seems.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16240",
      "postDate": "10/26/2012 11:03:15",
      "content": "<p>[quote=Gábor Melis;16231]</p>\r\n<p>Can we make multiple submissions and select one later in this phase too? By the way, there leaderboard will not be updated until the submissions are closed, right?</p>\r\n<p>[/quote]You can make multiple submissions, but you won't receive any feedback on them prior to the end of the competition (beyond whether they were parsed and scored correctly). You're free to retrain on the October 9 data or to use your previously-trained\r\n model, as you see fit. However, this should be the only change that you make to your model, beyond minor bug fixes that are necessary to get your model to run on the new dataset.</p>\r\n<p>Minor bug fixes are OK, if, for example, one of the new questions breaks your feature extraction code. However, this should not include things that impacts model performance, such as modifying hyperparameters that aren't set automatically by your training\r\n script.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16241",
      "postDate": "10/26/2012 11:11:33",
      "content": "<p>[quote=jsn13;16235]</p>\r\n<p>train-sample_October_9_2012.csv apparently has a lot of &quot;nan&quot;s where the original train-sample.csv had empty CSV fields. So it's still either a preprocessing script or modifications to our parsers, it seems.</p>\r\n<p>[/quote]Thanks for pointing &nbsp;this out. This occurred because I modified one of my functions to use Pandas to do I/O after releasing the initial file, which defaults to NaN for empty fields in a column that is numeric otherwise.</p>\r\n<p>I've corrected this and uploaded train-sample_October_9_2012_v2.csv.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16272",
      "postDate": "10/27/2012 03:16:36",
      "content": "<p>[quote=Ben Hamner;16194]</p>\r\n<p>Final submissions currently remain disabled, and they should be enabled by the end of the day on October 25, once the fix goes live.</p>\r\n<p>[/quote]</p>\r\n<p>Hi Ben, any news on that?&nbsp;When you believe we are going to be able to make our final submissions?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16273",
      "postDate": "10/27/2012 06:29:09",
      "content": "<p>[quote=James Petterson;16272]</p>\r\n<p>[quote=Ben Hamner;16194]</p>\r\n<p>Final submissions currently remain disabled, and they should be enabled by the end of the day on October 25, once the fix goes live.</p>\r\n<p>[/quote]</p>\r\n<p>Hi Ben, any news on that?&nbsp;When you believe we are going to be able to make our final submissions?&nbsp;</p>\r\n<p>[/quote]This is enabled. You have 7 days - from now to the end of November 3, 2012 UTC to make your final submission.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16275",
      "postDate": "10/27/2012 08:16:05",
      "content": "<p>[quote=Ben Hamner;16273]</p>\r\n<p>This is enabled. You have 7 days - from now to the end of November 3, 2012 UTC to make your final submission.</p>\r\n<p>[/quote]</p>\r\n<p>Thanks Ben. There is, however, a bug - please see the attached screenshot, it seems all new submissions are getting a score of zero.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16276",
      "postDate": "10/27/2012 08:24:10",
      "content": "<p>[quote=James Petterson;16275]</p>\r\n<p>Thanks Ben. There is, however, a bug - please see the attached screenshot, it seems that all new submissions are getting a score of zero.</p>\r\n<p>[/quote]</p>\r\n<p>Another thing: shouldn't the 'your submissions' page show only my private leaderboard submissions from now on? It still shows my public leaderboard ones.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16311",
      "postDate": "10/28/2012 23:36:43",
      "content": "<p>Thanks Ben.</p>\r\n<p>In the Impermium competition, only those who submitted score on test were ranked. Rest did not receive a rank.<br>\r\nhowever while there were 118 participants, only 50 submitted and everybody got ranked out of 50,</p>\r\n<p>How will it be here - will folks be ranked amongst all those who submit OR among all those who had entered competition?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16331",
      "postDate": "10/29/2012 18:44:03",
      "content": "<p>[quote=James Petterson;16276]</p>\r\n<p>Another thing: shouldn't the 'your submissions' page show only my private leaderboard submissions from now on? It still shows my public leaderboard ones.</p>\r\n<p>[/quote]No, it should be showing both public &amp; private leaderboard submissions.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16332",
      "postDate": "10/29/2012 18:44:59",
      "content": "<p>[quote=James Petterson;16275]</p>\r\n<p>[quote=Ben Hamner;16273]</p>\r\n<p>This is enabled. You have 7 days - from now to the end of November 3, 2012 UTC to make your final submission.</p>\r\n<p>[/quote]</p>\r\n<p>Thanks Ben. There is, however, a bug - please see the attached screenshot, it seems all new submissions are getting a score of zero.</p>\r\n<p>[/quote]Thanks - we're aware this is a bit confusing (but it means you're submission was parsed correctly &amp; doesn't signify any material problem).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16333",
      "postDate": "10/29/2012 18:46:18",
      "content": "<p>[quote=Black Magic;16311]</p>\r\n<p>Thanks Ben.</p>\r\n<p>In the Impermium competition, only those who submitted score on test were ranked. Rest did not receive a rank.<br>\r\nhowever while there were 118 participants, only 50 submitted and everybody got ranked out of 50,</p>\r\n<p>How will it be here - will folks be ranked amongst all those who submit OR among all those who had entered competition?&nbsp;</p>\r\n<p>[/quote]Only those who make a final submission will be ranked (once the necessary changes get pushed to our codebase).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16334",
      "postDate": "10/29/2012 18:46:30",
      "content": "<p>Ben my friend,</p>\r\n<p>Please respond to my post</p>\r\n<p>&nbsp;</p>\r\n<p>Thanks<br>\r\nkiran&nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>[quote=Black Magic;16311]</p>\r\n<p>Thanks Ben.</p>\r\n<p>In the Impermium competition, only those who submitted score on test were ranked. Rest did not receive a rank.<br>\r\nhowever while there were 118 participants, only 50 submitted and everybody got ranked out of 50,</p>\r\n<p>How will it be here - will folks be ranked amongst all those who submit OR among all those who had entered competition?&nbsp;</p>\r\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16372",
      "postDate": "10/31/2012 00:10:09",
      "content": "<p>I am also interested in the answer to this question.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 16202,
      "author_name": "zygmunt",
      "author_url": "",
      "post_date": "10/25/2012 14:13:34",
      "content": "<p>Call me picky, but I'd welcome you using a more standard notation for dates, instead of 10/9 or 11/3 - not all contestants are American. In fact, when you look at the TOP 30 users, only a third are. And half of them have names that don't sound native at\r\n all.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16220,
      "author_name": "",
      "author_url": "",
      "post_date": "10/26/2012 02:13:18",
      "content": "<p>Hi Ben</p>\r\n<p>Does &quot;<span>Raw TrainingSet</span>&quot; contain the new training data? I find that the &quot;CloseReason&quot; in many questions are empty. Could you have a look? Thanks! &nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16226,
      "author_name": "andysloane",
      "author_url": "",
      "post_date": "10/26/2012 05:09:14",
      "content": "<p>Yes, this new data set has a different schema than before. Instead of OpenStatus you have IsClosed and CloseReason. Do we need to update our parsers for the training set, then?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16227,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "10/26/2012 05:22:17",
      "content": "<p>[quote=Foxtrot;16202]</p>\r\n<p>Call me picky, but I'd welcome you using a more standard notation for dates, instead of 10/9 or 11/3 - not all contestants are American. In fact, when you look at the TOP 30 users, only a third are. And half of them have names that don't sound native at\r\n all.</p>\r\n<p>[/quote]Thanks for pointing this out - I have updated my post accordingly and will remember this for the future. Have asked my team to do the same.</p>\r\n<p>I lived in Switzerland for a year &amp; know it's confusing seeing 11/3 and wondering if it means November 3 or March 11.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16228,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "10/26/2012 05:27:51",
      "content": "<p>[quote=Erheng Zhong;16220]</p>\r\n<p>Hi Ben</p>\r\n<p>Does &quot;<span>Raw TrainingSet</span>&quot; contain the new training data? I find that the &quot;CloseReason&quot; in many questions are empty. Could you have a look? Thanks! &nbsp;</p>\r\n<p>[/quote]That was the format StackOverflow had originally provided the data in. I'd modified it to make it slightly easier to work with (so the ground truth was in a single column). The reformatted file is up as train_October_9_2012.csv. I'm currently travelling\r\n and on a flaky hotel internet connection, but other compressed formats of the same file will be appearing in the next 6 hours.</p>\r\n<p>Additionally, I've recreated the stratified sample from the new training set (train-sample_October_9_2012.csv) in case anyone's finals models used the subsampled set.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16229,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "10/26/2012 05:29:03",
      "content": "<p>[quote=Andy Sloane;16226]</p>\r\n<p>Yes, this new data set has a different schema than before. Instead of OpenStatus you have IsClosed and CloseReason. Do we need to update our parsers for the training set, then?</p>\r\n<p>[/quote]No, use train_October_9_2012.csv (just went live on the data page)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16231,
      "author_name": "melisgl",
      "author_url": "",
      "post_date": "10/26/2012 07:03:21",
      "content": "<p>Can we make multiple submissions and select one later in this phase too? By the way, there leaderboard will not be updated until the submissions are closed, right?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16232,
      "author_name": "melisgl",
      "author_url": "",
      "post_date": "10/26/2012 07:43:33",
      "content": "<p>Replying to myself, my original train of thought was that I may decide to submit the results from the model that was trained on the data up to august, if I see that the distribution of classes is more similar that way. But since that goes against the spirit\r\n of the competition, I'll submit the results of the version retrained on the oct9 dataset.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16235,
      "author_name": "jsn1313",
      "author_url": "",
      "post_date": "10/26/2012 08:52:49",
      "content": "<p>[quote=Ben Hamner;16228]</p>\r\n<p>Additionally, I've recreated the stratified sample from the new training set (train-sample_October_9_2012.csv) in case anyone's finals models used the subsampled set.</p>\r\n<p>[/quote]</p>\r\n<p>train-sample_October_9_2012.csv apparently has a lot of &quot;nan&quot;s where the original train-sample.csv had empty CSV fields. So it's still either a preprocessing script or modifications to our parsers, it seems.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16240,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "10/26/2012 11:03:15",
      "content": "<p>[quote=Gábor Melis;16231]</p>\r\n<p>Can we make multiple submissions and select one later in this phase too? By the way, there leaderboard will not be updated until the submissions are closed, right?</p>\r\n<p>[/quote]You can make multiple submissions, but you won't receive any feedback on them prior to the end of the competition (beyond whether they were parsed and scored correctly). You're free to retrain on the October 9 data or to use your previously-trained\r\n model, as you see fit. However, this should be the only change that you make to your model, beyond minor bug fixes that are necessary to get your model to run on the new dataset.</p>\r\n<p>Minor bug fixes are OK, if, for example, one of the new questions breaks your feature extraction code. However, this should not include things that impacts model performance, such as modifying hyperparameters that aren't set automatically by your training\r\n script.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16241,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "10/26/2012 11:11:33",
      "content": "<p>[quote=jsn13;16235]</p>\r\n<p>train-sample_October_9_2012.csv apparently has a lot of &quot;nan&quot;s where the original train-sample.csv had empty CSV fields. So it's still either a preprocessing script or modifications to our parsers, it seems.</p>\r\n<p>[/quote]Thanks for pointing &nbsp;this out. This occurred because I modified one of my functions to use Pandas to do I/O after releasing the initial file, which defaults to NaN for empty fields in a column that is numeric otherwise.</p>\r\n<p>I've corrected this and uploaded train-sample_October_9_2012_v2.csv.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16272,
      "author_name": "jpetterson",
      "author_url": "",
      "post_date": "10/27/2012 03:16:36",
      "content": "<p>[quote=Ben Hamner;16194]</p>\r\n<p>Final submissions currently remain disabled, and they should be enabled by the end of the day on October 25, once the fix goes live.</p>\r\n<p>[/quote]</p>\r\n<p>Hi Ben, any news on that?&nbsp;When you believe we are going to be able to make our final submissions?&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16273,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "10/27/2012 06:29:09",
      "content": "<p>[quote=James Petterson;16272]</p>\r\n<p>[quote=Ben Hamner;16194]</p>\r\n<p>Final submissions currently remain disabled, and they should be enabled by the end of the day on October 25, once the fix goes live.</p>\r\n<p>[/quote]</p>\r\n<p>Hi Ben, any news on that?&nbsp;When you believe we are going to be able to make our final submissions?&nbsp;</p>\r\n<p>[/quote]This is enabled. You have 7 days - from now to the end of November 3, 2012 UTC to make your final submission.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16275,
      "author_name": "jpetterson",
      "author_url": "",
      "post_date": "10/27/2012 08:16:05",
      "content": "<p>[quote=Ben Hamner;16273]</p>\r\n<p>This is enabled. You have 7 days - from now to the end of November 3, 2012 UTC to make your final submission.</p>\r\n<p>[/quote]</p>\r\n<p>Thanks Ben. There is, however, a bug - please see the attached screenshot, it seems all new submissions are getting a score of zero.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16276,
      "author_name": "jpetterson",
      "author_url": "",
      "post_date": "10/27/2012 08:24:10",
      "content": "<p>[quote=James Petterson;16275]</p>\r\n<p>Thanks Ben. There is, however, a bug - please see the attached screenshot, it seems that all new submissions are getting a score of zero.</p>\r\n<p>[/quote]</p>\r\n<p>Another thing: shouldn't the 'your submissions' page show only my private leaderboard submissions from now on? It still shows my public leaderboard ones.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16311,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "10/28/2012 23:36:43",
      "content": "<p>Thanks Ben.</p>\r\n<p>In the Impermium competition, only those who submitted score on test were ranked. Rest did not receive a rank.<br>\r\nhowever while there were 118 participants, only 50 submitted and everybody got ranked out of 50,</p>\r\n<p>How will it be here - will folks be ranked amongst all those who submit OR among all those who had entered competition?&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16331,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "10/29/2012 18:44:03",
      "content": "<p>[quote=James Petterson;16276]</p>\r\n<p>Another thing: shouldn't the 'your submissions' page show only my private leaderboard submissions from now on? It still shows my public leaderboard ones.</p>\r\n<p>[/quote]No, it should be showing both public &amp; private leaderboard submissions.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16332,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "10/29/2012 18:44:59",
      "content": "<p>[quote=James Petterson;16275]</p>\r\n<p>[quote=Ben Hamner;16273]</p>\r\n<p>This is enabled. You have 7 days - from now to the end of November 3, 2012 UTC to make your final submission.</p>\r\n<p>[/quote]</p>\r\n<p>Thanks Ben. There is, however, a bug - please see the attached screenshot, it seems all new submissions are getting a score of zero.</p>\r\n<p>[/quote]Thanks - we're aware this is a bit confusing (but it means you're submission was parsed correctly &amp; doesn't signify any material problem).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16333,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "10/29/2012 18:46:18",
      "content": "<p>[quote=Black Magic;16311]</p>\r\n<p>Thanks Ben.</p>\r\n<p>In the Impermium competition, only those who submitted score on test were ranked. Rest did not receive a rank.<br>\r\nhowever while there were 118 participants, only 50 submitted and everybody got ranked out of 50,</p>\r\n<p>How will it be here - will folks be ranked amongst all those who submit OR among all those who had entered competition?&nbsp;</p>\r\n<p>[/quote]Only those who make a final submission will be ranked (once the necessary changes get pushed to our codebase).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16334,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "10/29/2012 18:46:30",
      "content": "<p>Ben my friend,</p>\r\n<p>Please respond to my post</p>\r\n<p>&nbsp;</p>\r\n<p>Thanks<br>\r\nkiran&nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>[quote=Black Magic;16311]</p>\r\n<p>Thanks Ben.</p>\r\n<p>In the Impermium competition, only those who submitted score on test were ranked. Rest did not receive a rank.<br>\r\nhowever while there were 118 participants, only 50 submitted and everybody got ranked out of 50,</p>\r\n<p>How will it be here - will folks be ranked amongst all those who submit OR among all those who had entered competition?&nbsp;</p>\r\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16372,
      "author_name": "cdubois",
      "author_url": "",
      "post_date": "10/31/2012 00:10:09",
      "content": "<p>I am also interested in the answer to this question.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "16194": "",
    "16202": "",
    "16220": "",
    "16226": "",
    "16227": "",
    "16228": "",
    "16229": "",
    "16231": "",
    "16232": "",
    "16235": "",
    "16240": "",
    "16241": "",
    "16272": "",
    "16273": "",
    "16275": "",
    "16276": "",
    "16311": "",
    "16331": "",
    "16332": "",
    "16333": "",
    "16334": "",
    "16372": ""
  },
  "source": "meta"
}