{ "id": 44434, "title": "Model Upload is now enabled", "url": "/competitions/passenger-screening-algorithm-challenge/discussion/44434", "author_name": "", "post_date": "2017-11-28T16:59:29.010480200Z", "votes": 1, "comment_count": 42, "views": 0, "content": "

All,

\n\n

The Team Model Upload has been enabled. You have until Dec. 10 at 23:59 UTC to upload your model for Stage 2. (use Team tab: here).

\n\n

See https://www.kaggle.com/two-stage-faq as a resource for all your questions.

\n\n

Thanks!

", "messages": [ { "id": "249523", "postDate": "11/28/2017 16:59:29", "content": "

All,

\n\n

The Team Model Upload has been enabled. You have until Dec. 10 at 23:59 UTC to upload your model for Stage 2. (use Team tab: here).

\n\n

See https://www.kaggle.com/two-stage-faq as a resource for all your questions.

\n\n

Thanks!

", "rawMarkdown": "All,\n\nThe Team Model Upload has been enabled. You have until Dec. 10 at 23:59 UTC to upload your model for Stage 2. (use Team tab: [here][1]).\n\nSee https://www.kaggle.com/two-stage-faq as a resource for all your questions.\n\nThanks!\n\n\n [1]: https://www.kaggle.com/c/passenger-screening-algorithm-challenge/team", "votes": null }, { "id": "249933", "postDate": "11/29/2017 14:07:23", "content": "

I assume you mean until 23:59 UTC on Dec 10th.

", "rawMarkdown": "I assume you mean until 23:59 UTC on Dec 10th.", "votes": null }, { "id": "249946", "postDate": "11/29/2017 14:39:33", "content": "

That's correct - thanks. Edited the post above for clarification.

", "rawMarkdown": "That's correct - thanks. Edited the post above for clarification.", "votes": null }, { "id": "250011", "postDate": "11/29/2017 16:26:56", "content": "

Thanks; I hate to be a bother, but I've been wondering about something: do you folks really accept encrypted archives and MD5 hashes in place of models? I was under the impression that MD5 collisions are pretty easy to arrange these days?

", "rawMarkdown": "Thanks; I hate to be a bother, but I've been wondering about something: do you folks really accept encrypted archives and MD5 hashes in place of models? I was under the impression that MD5 collisions are pretty easy to arrange these days?", "votes": null }, { "id": "250025", "postDate": "11/29/2017 16:50:55", "content": "

Thanks for bringing this up. That documentation was created when creating MD5 collisions was less practical/possible than it is now. I've updated the page to say SHA256.

", "rawMarkdown": "Thanks for bringing this up. That documentation was created when creating MD5 collisions was less practical/possible than it is now. I've updated the page to say SHA256.", "votes": null }, { "id": "250045", "postDate": "11/29/2017 17:17:09", "content": "

Wow, that was fast! Thanks William :)

", "rawMarkdown": "Wow, that was fast! Thanks William :)", "votes": null }, { "id": "250198", "postDate": "11/29/2017 23:14:17", "content": "

Ok I am confused. On the rules page it says:

\n\n

Model Upload & First Stage Deadline: 12/15/2017 11:59 PM UTC

\n\n

Second Stage Start: 12/11/2017 11:59 PM UTC

\n\n

Final Submission Deadline: 12/15/2017 11:59 PM UTC

\n\n

So we have 3 dates now for model upload. And the rules pages shows the second stage starting after the date of the model upload/first stage deadline.

\n\n

Can you please clarify and make the doc pages consistent with the clarification.

\n\n

Brian.

", "rawMarkdown": "Ok I am confused. On the rules page it says:\n\nModel Upload & First Stage Deadline: 12/15/2017 11:59 PM UTC\n\nSecond Stage Start: 12/11/2017 11:59 PM UTC\n\nFinal Submission Deadline: 12/15/2017 11:59 PM UTC\n\n\nSo we have 3 dates now for model upload. And the rules pages shows the second stage starting after the date of the model upload/first stage deadline.\n\nCan you please clarify and make the doc pages consistent with the clarification.\n\nBrian.", "votes": null }, { "id": "250475", "postDate": "11/30/2017 03:32:30", "content": "

Hi Brian,

\n\n

The model upload and first stage deadline is 12/10 - I've updated the rules to correct the typo. The Timeline tab has been accurate and supercedes.

\n\n

Please read the FAQ page in the above post regarding how the two stage competitions work regarding model uploads and beginning/end of stages.

\n\n

Thanks!

", "rawMarkdown": "Hi Brian,\n\nThe model upload and first stage deadline is 12/10 - I've updated the rules to correct the typo. The Timeline tab has been accurate and supercedes.\n\nPlease read the FAQ page in the above post regarding how the two stage competitions work regarding model uploads and beginning/end of stages.\n\nThanks!", "votes": null }, { "id": "250514", "postDate": "11/30/2017 04:22:05", "content": "

I am good with the two stage. I thought the dates were odd on the rules page. Meant to ask before. Never noticed the timeline tab until now. In any event, thanks for clarifying.

\n\n

Brian.

", "rawMarkdown": "I am good with the two stage. I thought the dates were odd on the rules page. Meant to ask before. Never noticed the timeline tab until now. In any event, thanks for clarifying.\n\nBrian.", "votes": null }, { "id": "251334", "postDate": "12/01/2017 02:35:34", "content": "

Just for the sake of clarity, you don't want us to actually upload the models themselves, you just want us to upload the code responsible for producing the models. Even though RNG means that executing the code again may produce slightly different results on subsequent runs?

", "rawMarkdown": "Just for the sake of clarity, you don't want us to actually upload the models themselves, you just want us to upload the code responsible for producing the models. Even though RNG means that executing the code again may produce slightly different results on subsequent runs?", "votes": null }, { "id": "251731", "postDate": "12/01/2017 16:03:51", "content": "

Per the FAQ: \"In order to achieve this, we ask that you upload your source code, including the correct parameters that you used for generating your submission files\" You can set the random seed.

", "rawMarkdown": "Per the FAQ: \"In order to achieve this, we ask that you upload your source code, including the correct parameters that you used for generating your submission files\" You can set the random seed.", "votes": null }, { "id": "251779", "postDate": "12/01/2017 17:55:49", "content": "

I reread the stage2 FAQ. To clarify your follow up statement: all we need to do is submit a program that when executed generates our submission csv file? The method of training or discovering such a program does not itself have to be submitted. In other words, we just need to submit end-to-end inference code, and not any training code?

", "rawMarkdown": "I reread the stage2 FAQ. To clarify your follow up statement: all we need to do is submit a program that when executed generates our submission csv file? The method of training or discovering such a program does not itself have to be submitted. In other words, we just need to submit end-to-end **inference code**, and not any training code?", "votes": null }, { "id": "251786", "postDate": "12/01/2017 18:05:58", "content": "

Training code is also required.

", "rawMarkdown": "Training code is also required.", "votes": null }, { "id": "251809", "postDate": "12/01/2017 18:37:55", "content": "

For the purpose of confirming that our submission matches our model, will you be executing our inference program, or retraining everything from scratch and then running the inference program? When doing inference, I plan on loading model checkpoints to avoid retraining.

\n\n

My main concern is that there are known non-deterministic ops in tensorflow that are used in calculating gradients (training): https://github.com/tensorflow/tensorflow/issues/2652, https://github.com/tensorflow/tensorflow/issues/2732, https://github.com/tensorflow/tensorflow/issues/3103

\n\n

I'm currently using tensorflow 1.3. As one of the comments pointed out, 1.4 might have solved this issue for some reductions but it's not guaranteed for all training ops.

\n\n

My concern is that retraining might result in slightly different weights yielding slightly different predictions. However this is not an issue if you are only checking the final inference program.

", "rawMarkdown": "For the purpose of confirming that our submission matches our model, will you be executing our inference program, or retraining everything from scratch and then running the inference program? When doing inference, I plan on loading model checkpoints to avoid retraining.\n\nMy main concern is that there are known non-deterministic ops in tensorflow that are used in calculating gradients (training): https://github.com/tensorflow/tensorflow/issues/2652, https://github.com/tensorflow/tensorflow/issues/2732, https://github.com/tensorflow/tensorflow/issues/3103\n\nI'm currently using tensorflow 1.3. As one of the comments pointed out, 1.4 might have solved this issue for some reductions but it's not guaranteed for all training ops. \n\nMy concern is that retraining might result in slightly different weights yielding slightly different predictions. However this is not an issue if you are only checking the final inference program.", "votes": null }, { "id": "251937", "postDate": "12/01/2017 23:11:20", "content": "

I'm just a contestant, but the impression I've gotten is that basically your submission will be reviewed \"in good faith.\" I'm sure they'll take your word for it and do their best to recreate your score; if necessary, they'll probably figure something out on-the-fly (in the case of a tie and utter failure in recreating a submission's predictions or something.)

\n\n

After reviewing the two-stage FAQ again (it's been a few weeks), I guess my point is that you should just upload the weights and largely bypass all of these concerns. I don't see any disadvantage to that, personally. Since the upload of models ends prior to the final test set being released, I doubt they'll be retraining models unless necessary. Other than pulling in \"altered\" weights under the guise of using a mostly pre-trained architecture, I don't see any reason to be suspicious of someone's uploaded model.

\n\n

Am I missing something?

", "rawMarkdown": "I'm just a contestant, but the impression I've gotten is that basically your submission will be reviewed \"in good faith.\" I'm sure they'll take your word for it and do their best to recreate your score; if necessary, they'll probably figure something out on-the-fly (in the case of a tie and utter failure in recreating a submission's predictions or something.)\n\nAfter reviewing the two-stage FAQ again (it's been a few weeks), I guess my point is that you should just upload the weights and largely bypass all of these concerns. I don't see any disadvantage to that, personally. Since the upload of models ends prior to the final test set being released, I doubt they'll be retraining models unless necessary. Other than pulling in \"altered\" weights under the guise of using a mostly pre-trained architecture, I don't see any reason to be suspicious of someone's uploaded model.\n\nAm I missing something?", "votes": null }, { "id": "252043", "postDate": "12/02/2017 05:52:52", "content": "

How should we submit permitted corrections for stage 2 (for example, changed paths to input data)? Should we submit weights for the model for both selected submissions of the stage 1?

", "rawMarkdown": "How should we submit permitted corrections for stage 2 (for example, changed paths to input data)? Should we submit weights for the model for both selected submissions of the stage 1?", "votes": null }, { "id": "252059", "postDate": "12/02/2017 06:16:50", "content": "

What is tested size limit for model upload? My model is about 50GB and I'm not sure if I can successfully upload it as single file using cable modem.

", "rawMarkdown": "What is tested size limit for model upload? My model is about 50GB and I'm not sure if I can successfully upload it as single file using cable modem.", "votes": null }, { "id": "252453", "postDate": "12/03/2017 00:17:26", "content": "

By the nature of things, different random seeds, could, in principle, produce minor differences in the final result, so perhaps they'll rerun the model, if it doesn't produce a result consistent with your final prediction? And if they really can't reproduce it all, then there might be trouble, I imagine.

", "rawMarkdown": "By the nature of things, different random seeds, could, in principle, produce minor differences in the final result, so perhaps they'll rerun the model, if it doesn't produce a result consistent with your final prediction? And if they really can't reproduce it all, then there might be trouble, I imagine.", "votes": null }, { "id": "252456", "postDate": "12/03/2017 00:24:39", "content": "

Cable Modem? Is that a feature in gsutil?

", "rawMarkdown": "Cable Modem? Is that a feature in gsutil?", "votes": null }, { "id": "252457", "postDate": "12/03/2017 00:25:59", "content": "

I use powerful personal computer for computation. I will use checksums and exclude optional data.

", "rawMarkdown": "I use powerful personal computer for computation. I will use checksums and exclude optional data.", "votes": null }, { "id": "252630", "postDate": "12/03/2017 12:12:44", "content": "

I have literally zero experience with the kind of cloud computing platform they'd be using, but I'd wager Kaggle has access to development environments that would eat your powerful machine for breakfast. I don't think it's a concern, but I may be wrong :)

", "rawMarkdown": "I have literally zero experience with the kind of cloud computing platform they'd be using, but I'd wager Kaggle has access to development environments that would eat your powerful machine for breakfast. I don't think it's a concern, but I may be wrong :)", "votes": null }, { "id": "252788", "postDate": "12/03/2017 21:52:32", "content": "

Addison,\ncan you clarify the Dec4 deadline and confirm that models developed and trained entirely from Stage1 data need not be posted until Dec 10

", "rawMarkdown": "Addison,\ncan you clarify the Dec4 deadline and confirm that models developed and trained entirely from Stage1 data need not be posted until Dec 10", "votes": null }, { "id": "252846", "postDate": "12/04/2017 02:22:37", "content": "

That is correct. The Dec 4 deadline is to post your pre-trained models and external data used to the \"official external data\" thread. You then have until Dec 10 to upload your model.

", "rawMarkdown": "That is correct. The Dec 4 deadline is to post your pre-trained models and external data used to the \"official external data\" thread. You then have until Dec 10 to upload your model.", "votes": null }, { "id": "253919", "postDate": "12/05/2017 21:30:29", "content": "

I have a question.

\n\n

Throughout the competition, I've been improving my model using hyper-parameter optimization. I know that this is allowed by the rules as long as it's entirely automated, and it is. But this exploration is very time consuming and I've spent weeks or months training different models. So my question is, can I use the model I've already tuned (the same model I've used in stage 1) for stage 2 for prediction? Even though I can't retrain from scratch in the allotted time, my code and instructions would theoretically be able to produce the same model or something reasonably close to it. I would be submitting the model I've already trained so the exact same submission can be reproduced very quickly.

\n\n

To simplify my question: Do I actually have to retrain everything from scratch, which might be impossible for me in the 5-4 days we have, or can I use the same model I used in stage 1, knowing that that model can be recreated from the code given more time?

", "rawMarkdown": "I have a question.\n\nThroughout the competition, I've been improving my model using hyper-parameter optimization. I know that this is allowed by the rules as long as it's entirely automated, and it is. But this exploration is very time consuming and I've spent weeks or months training different models. So my question is, can I use the model I've already tuned (the same model I've used in stage 1) for stage 2 for prediction? Even though I can't retrain from scratch in the allotted time, my code and instructions would theoretically be able to produce the same model or something reasonably close to it. I would be submitting the model I've already trained so the exact same submission can be reproduced very quickly.\n\nTo simplify my question: Do I actually have to retrain everything from scratch, which might be impossible for me in the 5-4 days we have, or can I use the same model I used in stage 1, knowing that that model can be recreated from the code given more time?", "votes": null }, { "id": "253925", "postDate": "12/05/2017 21:41:26", "content": "

You can't use your scores even for automated optimization. If you don't use them, then your stage 1 model wouldn't need to be optimized again for the stage 2.

", "rawMarkdown": "You can't use your scores even for automated optimization. If you don't use them, then your stage 1 model wouldn't need to be optimized again for the stage 2.", "votes": null }, { "id": "253933", "postDate": "12/05/2017 21:52:04", "content": "

I'm not using the validation set for the optimization, I'm using a split of the training set.

", "rawMarkdown": "I'm not using the validation set for the optimization, I'm using a split of the training set.", "votes": null }, { "id": "253960", "postDate": "12/05/2017 23:15:18", "content": "

Hi Kevin, \nMy understanding is that you do not need to retrain everything from scratch for stage 2, you can just use your existing model (trained weights) to predict directly on stage 2 once its released. In fact, I don't think you are allowed to change your model at all after its submitted for stage 1 but you would have the option to re-train (or fine-tune) with stage 1 test images/ground truth labels included during stage 2 since those labels will also be released for stage 2.

", "rawMarkdown": "Hi Kevin, \nMy understanding is that you do not need to retrain everything from scratch for stage 2, you can just use your existing model (trained weights) to predict directly on stage 2 once its released. In fact, I don't think you are allowed to change your model at all after its submitted for stage 1 but you would have the option to re-train (or fine-tune) with stage 1 test images/ground truth labels included during stage 2 since those labels will also be released for stage 2.", "votes": null }, { "id": "254020", "postDate": "12/06/2017 03:38:39", "content": "

I wonder why we would have to publish external data sources if it would be allowed to package unverifiable weights files.

\n\n

Also, per Addison: \nPer the FAQ: \"In order to achieve this, we ask that you upload your source code, including the correct parameters that you used for generating your submission files\"

", "rawMarkdown": "I wonder why we would have to publish external data sources if it would be allowed to package unverifiable weights files.\n\nAlso, per Addison: \nPer the FAQ: \"In order to achieve this, we ask that you upload your source code, including the correct parameters that you used for generating your submission files\"", "votes": null }, { "id": "254072", "postDate": "12/06/2017 06:07:16", "content": "

I think the reason for posting external data is to not give any team an unfair data advantage. For example, if you're a data science company that has some proprietary extra data or models pre-trained on that extra data, it would be unfair to the other competitors unless that data was made publicly available to everyone. That's why you're required to post all your data/model sources.

\n\n

In contrast, my weights came entirely from competition data and are verifiable, just not in the amount of time we have for stage 2, which is my main concern.

", "rawMarkdown": "I think the reason for posting external data is to not give any team an unfair **data** advantage. For example, if you're a data science company that has some proprietary extra data or models pre-trained on that extra data, it would be unfair to the other competitors unless that data was made publicly available to everyone. That's why you're required to post all your data/model sources.\n\nIn contrast, my weights came entirely from competition data and are verifiable, just not in the amount of time we have for stage 2, which is my main concern.", "votes": null }, { "id": "255698", "postDate": "12/09/2017 20:44:16", "content": "

Please provide official answer with clarification for the questions there: https://www.kaggle.com/c/passenger-screening-algorithm-challenge/discussion/45353

", "rawMarkdown": "Please provide official answer with clarification for the questions there: https://www.kaggle.com/c/passenger-screening-algorithm-challenge/discussion/45353", "votes": null }, { "id": "255782", "postDate": "12/10/2017 04:09:09", "content": "

why I can not see my uploaded model after I submitted? how can I confirm that my model has been submitted?

", "rawMarkdown": "why I can not see my uploaded model after I submitted? how can I confirm that my model has been submitted?", "votes": null }, { "id": "255792", "postDate": "12/10/2017 05:15:50", "content": "

You should be able to see your model under Team > Your Model > Download. If you don't see it there, you should probably try to re-upload it.

", "rawMarkdown": "You should be able to see your model under Team > Your Model > Download. If you don't see it there, you should probably try to re-upload it.", "votes": null }, { "id": "255983", "postDate": "12/10/2017 20:27:19", "content": "

And.. you have to reload the page, because the page automatically loaded doesn't know about your model,.... tsssk Kaggle, you got me scared for a minute!

", "rawMarkdown": "And.. you have to reload the page, because the page automatically loaded doesn't know about your model,.... tsssk Kaggle, you got me scared for a minute!", "votes": null }, { "id": "256008", "postDate": "12/10/2017 23:11:55", "content": "
\n

Wensu wrote

\n \n
\n

why I can not see my uploaded model after I submitted? how can I confirm that my model has been submitted?

\n
\n
\n\n

The same happened to me: my first upload was visible in the \"preview\", but disappeared within minutes.

\n\n

Anyway, this is my model's SHA256: f3e38d9eb1211593b2aefe842d69dce41f3d92f2295d7757fc8243b06878d61d in case this happens again.

", "rawMarkdown": "> **Wensu wrote**\n> \n> > why I can not see my uploaded model after I submitted? how can I confirm that my model has been submitted?\n\nThe same happened to me: my first upload was visible in the \"preview\", but disappeared within minutes.\n\nAnyway, this is my model's SHA256: f3e38d9eb1211593b2aefe842d69dce41f3d92f2295d7757fc8243b06878d61d in case this happens again.", "votes": null }, { "id": "256012", "postDate": "12/10/2017 23:32:35", "content": "

On second thought, it's definitely a refresh issue in the browser (I updated my model again, but it was still showing the old one in the preview)

", "rawMarkdown": "On second thought, it's definitely a refresh issue in the browser (I updated my model again, but it was still showing the old one in the preview)", "votes": null }, { "id": "256017", "postDate": "12/11/2017 00:11:47", "content": "

I had problems with seeing correct uploaded file as well. So here is my SHA256, just in case:\nb3944fff04223a7db38e08f1cdc54d8ddf78c55ca58b23ad399684b1c29b8dbc

", "rawMarkdown": "I had problems with seeing correct uploaded file as well. So here is my SHA256, just in case:\nb3944fff04223a7db38e08f1cdc54d8ddf78c55ca58b23ad399684b1c29b8dbc", "votes": null }, { "id": "256026", "postDate": "12/11/2017 00:33:35", "content": "

Same here. Not able to see uploaded models. here is my SHA256, just in case.\nfb2c531b20b5a4e6911f49cb120fbbaa6c2743b568dee8cf60468e134909dcfc

", "rawMarkdown": "Same here. Not able to see uploaded models. here is my SHA256, just in case.\nfb2c531b20b5a4e6911f49cb120fbbaa6c2743b568dee8cf60468e134909dcfc", "votes": null }, { "id": "256035", "postDate": "12/11/2017 01:02:21", "content": "

I don't see where my model is uploaded either ¯_(ツ)_/¯

\n\n

SHA256: 24d30810ccccfc0fb042cf5bcd20aaa76418d8c2481dc6ab8d8b251bf5ea44a2

", "rawMarkdown": "I don't see where my model is uploaded either ¯\\_(ツ)_/¯\n\nSHA256: 24d30810ccccfc0fb042cf5bcd20aaa76418d8c2481dc6ab8d8b251bf5ea44a2", "votes": null }, { "id": "256049", "postDate": "12/11/2017 02:01:28", "content": "

It seemed to work just fine for me but now nothing shows anymore when I visit the \"team\" page. \nIs it normal to not see my submission listed anymore? (now without the \"Upload New File\" button of course).\nAs I also didn't get a receipt confirmation e-mail, just to make sure: the SHA256 for my submission file is: 841b48023fdaca68fab9aa480d3c4335f7478f2dcf86792c1848f017501080e1

", "rawMarkdown": "It seemed to work just fine for me but now nothing shows anymore when I visit the \"team\" page. \nIs it normal to not see my submission listed anymore? (now without the \"Upload New File\" button of course).\nAs I also didn't get a receipt confirmation e-mail, just to make sure: the SHA256 for my submission file is: 841b48023fdaca68fab9aa480d3c4335f7478f2dcf86792c1848f017501080e1", "votes": null }, { "id": "256051", "postDate": "12/11/2017 02:03:46", "content": "

I also don't see my model anymore, dropping sha just to make sure: 64a88bb1d1e5ea290cfcdf2d481c637b089ce8fa194c52f97143a7128bff440b

", "rawMarkdown": "I also don't see my model anymore, dropping sha just to make sure: 64a88bb1d1e5ea290cfcdf2d481c637b089ce8fa194c52f97143a7128bff440b", "votes": null }, { "id": "256060", "postDate": "12/11/2017 02:36:56", "content": "

I'm pretty sure this is normal since it seems to have affected everybody. But I'll post my SHA256 too just in case...\nA6AE6D40CA3F5D3191A0BA03ABBA9042A20E8D62C6032DA465493E84ECCE26A7

", "rawMarkdown": "I'm pretty sure this is normal since it seems to have affected everybody. But I'll post my SHA256 too just in case...\nA6AE6D40CA3F5D3191A0BA03ABBA9042A20E8D62C6032DA465493E84ECCE26A7", "votes": null }, { "id": "256404", "postDate": "12/11/2017 22:01:14", "content": "

The sha 256 hash for team TomaDon submission file is:\n78d5b48adaed987ea954fe43a15bbc0f130df8768f242125deaea6b5a74ead0a

", "rawMarkdown": "The sha 256 hash for team TomaDon submission file is:\n78d5b48adaed987ea954fe43a15bbc0f130df8768f242125deaea6b5a74ead0a", "votes": null }, { "id": "256542", "postDate": "12/12/2017 07:25:43", "content": "

to be on the safe side. 2832290d32b72074018fc21485731395f4a1d00778466ae498476fc1df7f21ca model_submission_1512486248.tar.gz

", "rawMarkdown": "to be on the safe side. 2832290d32b72074018fc21485731395f4a1d00778466ae498476fc1df7f21ca model_submission_1512486248.tar.gz", "votes": null } ], "comments": [ { "id": 249933, "author_name": "jduffy", "author_url": "", "post_date": "11/29/2017 14:07:23", "content": "

I assume you mean until 23:59 UTC on Dec 10th.

", "votes": null, "replies": [ { "id": 249946, "author_name": "addisonhoward", "author_url": "", "post_date": "11/29/2017 14:39:33", "content": "

That's correct - thanks. Edited the post above for clarification.

", "votes": null, "replies": [] } ] }, { "id": 250011, "author_name": "mmiron", "author_url": "", "post_date": "11/29/2017 16:26:56", "content": "

Thanks; I hate to be a bother, but I've been wondering about something: do you folks really accept encrypted archives and MD5 hashes in place of models? I was under the impression that MD5 collisions are pretty easy to arrange these days?

", "votes": null, "replies": [ { "id": 250025, "author_name": "wcukierski", "author_url": "", "post_date": "11/29/2017 16:50:55", "content": "

Thanks for bringing this up. That documentation was created when creating MD5 collisions was less practical/possible than it is now. I've updated the page to say SHA256.

", "votes": null, "replies": [] }, { "id": 250045, "author_name": "mmiron", "author_url": "", "post_date": "11/29/2017 17:17:09", "content": "

Wow, that was fast! Thanks William :)

", "votes": null, "replies": [] } ] }, { "id": 250198, "author_name": "jbfarrar", "author_url": "", "post_date": "11/29/2017 23:14:17", "content": "

Ok I am confused. On the rules page it says:

\n\n

Model Upload & First Stage Deadline: 12/15/2017 11:59 PM UTC

\n\n

Second Stage Start: 12/11/2017 11:59 PM UTC

\n\n

Final Submission Deadline: 12/15/2017 11:59 PM UTC

\n\n

So we have 3 dates now for model upload. And the rules pages shows the second stage starting after the date of the model upload/first stage deadline.

\n\n

Can you please clarify and make the doc pages consistent with the clarification.

\n\n

Brian.

", "votes": null, "replies": [ { "id": 250475, "author_name": "addisonhoward", "author_url": "", "post_date": "11/30/2017 03:32:30", "content": "

Hi Brian,

\n\n

The model upload and first stage deadline is 12/10 - I've updated the rules to correct the typo. The Timeline tab has been accurate and supercedes.

\n\n

Please read the FAQ page in the above post regarding how the two stage competitions work regarding model uploads and beginning/end of stages.

\n\n

Thanks!

", "votes": null, "replies": [] }, { "id": 250514, "author_name": "jbfarrar", "author_url": "", "post_date": "11/30/2017 04:22:05", "content": "

I am good with the two stage. I thought the dates were odd on the rules page. Meant to ask before. Never noticed the timeline tab until now. In any event, thanks for clarifying.

\n\n

Brian.

", "votes": null, "replies": [] } ] }, { "id": 251334, "author_name": "", "author_url": "", "post_date": "12/01/2017 02:35:34", "content": "

Just for the sake of clarity, you don't want us to actually upload the models themselves, you just want us to upload the code responsible for producing the models. Even though RNG means that executing the code again may produce slightly different results on subsequent runs?

", "votes": null, "replies": [ { "id": 251731, "author_name": "addisonhoward", "author_url": "", "post_date": "12/01/2017 16:03:51", "content": "

Per the FAQ: \"In order to achieve this, we ask that you upload your source code, including the correct parameters that you used for generating your submission files\" You can set the random seed.

", "votes": null, "replies": [] }, { "id": 251779, "author_name": "alondaks", "author_url": "", "post_date": "12/01/2017 17:55:49", "content": "

I reread the stage2 FAQ. To clarify your follow up statement: all we need to do is submit a program that when executed generates our submission csv file? The method of training or discovering such a program does not itself have to be submitted. In other words, we just need to submit end-to-end inference code, and not any training code?

", "votes": null, "replies": [] }, { "id": 251786, "author_name": "addisonhoward", "author_url": "", "post_date": "12/01/2017 18:05:58", "content": "

Training code is also required.

", "votes": null, "replies": [] }, { "id": 251809, "author_name": "alondaks", "author_url": "", "post_date": "12/01/2017 18:37:55", "content": "

For the purpose of confirming that our submission matches our model, will you be executing our inference program, or retraining everything from scratch and then running the inference program? When doing inference, I plan on loading model checkpoints to avoid retraining.

\n\n

My main concern is that there are known non-deterministic ops in tensorflow that are used in calculating gradients (training): https://github.com/tensorflow/tensorflow/issues/2652, https://github.com/tensorflow/tensorflow/issues/2732, https://github.com/tensorflow/tensorflow/issues/3103

\n\n

I'm currently using tensorflow 1.3. As one of the comments pointed out, 1.4 might have solved this issue for some reductions but it's not guaranteed for all training ops.

\n\n

My concern is that retraining might result in slightly different weights yielding slightly different predictions. However this is not an issue if you are only checking the final inference program.

", "votes": null, "replies": [] }, { "id": 251937, "author_name": "mmiron", "author_url": "", "post_date": "12/01/2017 23:11:20", "content": "

I'm just a contestant, but the impression I've gotten is that basically your submission will be reviewed \"in good faith.\" I'm sure they'll take your word for it and do their best to recreate your score; if necessary, they'll probably figure something out on-the-fly (in the case of a tie and utter failure in recreating a submission's predictions or something.)

\n\n

After reviewing the two-stage FAQ again (it's been a few weeks), I guess my point is that you should just upload the weights and largely bypass all of these concerns. I don't see any disadvantage to that, personally. Since the upload of models ends prior to the final test set being released, I doubt they'll be retraining models unless necessary. Other than pulling in \"altered\" weights under the guise of using a mostly pre-trained architecture, I don't see any reason to be suspicious of someone's uploaded model.

\n\n

Am I missing something?

", "votes": null, "replies": [] }, { "id": 252453, "author_name": "tothink", "author_url": "", "post_date": "12/03/2017 00:17:26", "content": "

By the nature of things, different random seeds, could, in principle, produce minor differences in the final result, so perhaps they'll rerun the model, if it doesn't produce a result consistent with your final prediction? And if they really can't reproduce it all, then there might be trouble, I imagine.

", "votes": null, "replies": [] }, { "id": 252788, "author_name": "lwalzer", "author_url": "", "post_date": "12/03/2017 21:52:32", "content": "

Addison,\ncan you clarify the Dec4 deadline and confirm that models developed and trained entirely from Stage1 data need not be posted until Dec 10

", "votes": null, "replies": [] }, { "id": 252846, "author_name": "addisonhoward", "author_url": "", "post_date": "12/04/2017 02:22:37", "content": "

That is correct. The Dec 4 deadline is to post your pre-trained models and external data used to the \"official external data\" thread. You then have until Dec 10 to upload your model.

", "votes": null, "replies": [] } ] }, { "id": 252043, "author_name": "dmitrykovba", "author_url": "", "post_date": "12/02/2017 05:52:52", "content": "

How should we submit permitted corrections for stage 2 (for example, changed paths to input data)? Should we submit weights for the model for both selected submissions of the stage 1?

", "votes": null, "replies": [] }, { "id": 252059, "author_name": "dmitrykovba", "author_url": "", "post_date": "12/02/2017 06:16:50", "content": "

What is tested size limit for model upload? My model is about 50GB and I'm not sure if I can successfully upload it as single file using cable modem.

", "votes": null, "replies": [ { "id": 252456, "author_name": "tothink", "author_url": "", "post_date": "12/03/2017 00:24:39", "content": "

Cable Modem? Is that a feature in gsutil?

", "votes": null, "replies": [] }, { "id": 252457, "author_name": "dmitrykovba", "author_url": "", "post_date": "12/03/2017 00:25:59", "content": "

I use powerful personal computer for computation. I will use checksums and exclude optional data.

", "votes": null, "replies": [] }, { "id": 252630, "author_name": "mmiron", "author_url": "", "post_date": "12/03/2017 12:12:44", "content": "

I have literally zero experience with the kind of cloud computing platform they'd be using, but I'd wager Kaggle has access to development environments that would eat your powerful machine for breakfast. I don't think it's a concern, but I may be wrong :)

", "votes": null, "replies": [] } ] }, { "id": 253919, "author_name": "hackerpoet", "author_url": "", "post_date": "12/05/2017 21:30:29", "content": "

I have a question.

\n\n

Throughout the competition, I've been improving my model using hyper-parameter optimization. I know that this is allowed by the rules as long as it's entirely automated, and it is. But this exploration is very time consuming and I've spent weeks or months training different models. So my question is, can I use the model I've already tuned (the same model I've used in stage 1) for stage 2 for prediction? Even though I can't retrain from scratch in the allotted time, my code and instructions would theoretically be able to produce the same model or something reasonably close to it. I would be submitting the model I've already trained so the exact same submission can be reproduced very quickly.

\n\n

To simplify my question: Do I actually have to retrain everything from scratch, which might be impossible for me in the 5-4 days we have, or can I use the same model I used in stage 1, knowing that that model can be recreated from the code given more time?

", "votes": null, "replies": [ { "id": 253925, "author_name": "dmitrykovba", "author_url": "", "post_date": "12/05/2017 21:41:26", "content": "

You can't use your scores even for automated optimization. If you don't use them, then your stage 1 model wouldn't need to be optimized again for the stage 2.

", "votes": null, "replies": [] }, { "id": 253933, "author_name": "hackerpoet", "author_url": "", "post_date": "12/05/2017 21:52:04", "content": "

I'm not using the validation set for the optimization, I'm using a split of the training set.

", "votes": null, "replies": [] }, { "id": 253960, "author_name": "jamesrequa", "author_url": "", "post_date": "12/05/2017 23:15:18", "content": "

Hi Kevin, \nMy understanding is that you do not need to retrain everything from scratch for stage 2, you can just use your existing model (trained weights) to predict directly on stage 2 once its released. In fact, I don't think you are allowed to change your model at all after its submitted for stage 1 but you would have the option to re-train (or fine-tune) with stage 1 test images/ground truth labels included during stage 2 since those labels will also be released for stage 2.

", "votes": null, "replies": [] }, { "id": 254020, "author_name": "bastiaanbergman", "author_url": "", "post_date": "12/06/2017 03:38:39", "content": "

I wonder why we would have to publish external data sources if it would be allowed to package unverifiable weights files.

\n\n

Also, per Addison: \nPer the FAQ: \"In order to achieve this, we ask that you upload your source code, including the correct parameters that you used for generating your submission files\"

", "votes": null, "replies": [] }, { "id": 254072, "author_name": "hackerpoet", "author_url": "", "post_date": "12/06/2017 06:07:16", "content": "

I think the reason for posting external data is to not give any team an unfair data advantage. For example, if you're a data science company that has some proprietary extra data or models pre-trained on that extra data, it would be unfair to the other competitors unless that data was made publicly available to everyone. That's why you're required to post all your data/model sources.

\n\n

In contrast, my weights came entirely from competition data and are verifiable, just not in the amount of time we have for stage 2, which is my main concern.

", "votes": null, "replies": [] } ] }, { "id": 255698, "author_name": "dmitrykovba", "author_url": "", "post_date": "12/09/2017 20:44:16", "content": "

Please provide official answer with clarification for the questions there: https://www.kaggle.com/c/passenger-screening-algorithm-challenge/discussion/45353

", "votes": null, "replies": [] }, { "id": 255782, "author_name": "w9wang2", "author_url": "", "post_date": "12/10/2017 04:09:09", "content": "

why I can not see my uploaded model after I submitted? how can I confirm that my model has been submitted?

", "votes": null, "replies": [ { "id": 255792, "author_name": "hackerpoet", "author_url": "", "post_date": "12/10/2017 05:15:50", "content": "

You should be able to see your model under Team > Your Model > Download. If you don't see it there, you should probably try to re-upload it.

", "votes": null, "replies": [] }, { "id": 255983, "author_name": "bastiaanbergman", "author_url": "", "post_date": "12/10/2017 20:27:19", "content": "

And.. you have to reload the page, because the page automatically loaded doesn't know about your model,.... tsssk Kaggle, you got me scared for a minute!

", "votes": null, "replies": [] }, { "id": 256008, "author_name": "olegtrott", "author_url": "", "post_date": "12/10/2017 23:11:55", "content": "
\n

Wensu wrote

\n \n
\n

why I can not see my uploaded model after I submitted? how can I confirm that my model has been submitted?

\n
\n
\n\n

The same happened to me: my first upload was visible in the \"preview\", but disappeared within minutes.

\n\n

Anyway, this is my model's SHA256: f3e38d9eb1211593b2aefe842d69dce41f3d92f2295d7757fc8243b06878d61d in case this happens again.

", "votes": null, "replies": [] }, { "id": 256012, "author_name": "olegtrott", "author_url": "", "post_date": "12/10/2017 23:32:35", "content": "

On second thought, it's definitely a refresh issue in the browser (I updated my model again, but it was still showing the old one in the preview)

", "votes": null, "replies": [] } ] }, { "id": 256017, "author_name": "serg14", "author_url": "", "post_date": "12/11/2017 00:11:47", "content": "

I had problems with seeing correct uploaded file as well. So here is my SHA256, just in case:\nb3944fff04223a7db38e08f1cdc54d8ddf78c55ca58b23ad399684b1c29b8dbc

", "votes": null, "replies": [ { "id": 256026, "author_name": "luckyguy", "author_url": "", "post_date": "12/11/2017 00:33:35", "content": "

Same here. Not able to see uploaded models. here is my SHA256, just in case.\nfb2c531b20b5a4e6911f49cb120fbbaa6c2743b568dee8cf60468e134909dcfc

", "votes": null, "replies": [] }, { "id": 256060, "author_name": "hackerpoet", "author_url": "", "post_date": "12/11/2017 02:36:56", "content": "

I'm pretty sure this is normal since it seems to have affected everybody. But I'll post my SHA256 too just in case...\nA6AE6D40CA3F5D3191A0BA03ABBA9042A20E8D62C6032DA465493E84ECCE26A7

", "votes": null, "replies": [] } ] }, { "id": 256035, "author_name": "jfaath", "author_url": "", "post_date": "12/11/2017 01:02:21", "content": "

I don't see where my model is uploaded either ¯_(ツ)_/¯

\n\n

SHA256: 24d30810ccccfc0fb042cf5bcd20aaa76418d8c2481dc6ab8d8b251bf5ea44a2

", "votes": null, "replies": [] }, { "id": 256049, "author_name": "gra55h0pper", "author_url": "", "post_date": "12/11/2017 02:01:28", "content": "

It seemed to work just fine for me but now nothing shows anymore when I visit the \"team\" page. \nIs it normal to not see my submission listed anymore? (now without the \"Upload New File\" button of course).\nAs I also didn't get a receipt confirmation e-mail, just to make sure: the SHA256 for my submission file is: 841b48023fdaca68fab9aa480d3c4335f7478f2dcf86792c1848f017501080e1

", "votes": null, "replies": [] }, { "id": 256051, "author_name": "despurious", "author_url": "", "post_date": "12/11/2017 02:03:46", "content": "

I also don't see my model anymore, dropping sha just to make sure: 64a88bb1d1e5ea290cfcdf2d481c637b089ce8fa194c52f97143a7128bff440b

", "votes": null, "replies": [] }, { "id": 256404, "author_name": "graydlpnwedu", "author_url": "", "post_date": "12/11/2017 22:01:14", "content": "

The sha 256 hash for team TomaDon submission file is:\n78d5b48adaed987ea954fe43a15bbc0f130df8768f242125deaea6b5a74ead0a

", "votes": null, "replies": [] }, { "id": 256542, "author_name": "ckomaki", "author_url": "", "post_date": "12/12/2017 07:25:43", "content": "

to be on the safe side. 2832290d32b72074018fc21485731395f4a1d00778466ae498476fc1df7f21ca model_submission_1512486248.tar.gz

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "249523": "All,\n\nThe Team Model Upload has been enabled. You have until Dec. 10 at 23:59 UTC to upload your model for Stage 2. (use Team tab: [here][1]).\n\nSee https://www.kaggle.com/two-stage-faq as a resource for all your questions.\n\nThanks!\n\n\n [1]: https://www.kaggle.com/c/passenger-screening-algorithm-challenge/team", "249933": "I assume you mean until 23:59 UTC on Dec 10th.", "249946": "That's correct - thanks. Edited the post above for clarification.", "250011": "Thanks; I hate to be a bother, but I've been wondering about something: do you folks really accept encrypted archives and MD5 hashes in place of models? I was under the impression that MD5 collisions are pretty easy to arrange these days?", "250025": "Thanks for bringing this up. That documentation was created when creating MD5 collisions was less practical/possible than it is now. I've updated the page to say SHA256.", "250045": "Wow, that was fast! Thanks William :)", "250198": "Ok I am confused. On the rules page it says:\n\nModel Upload & First Stage Deadline: 12/15/2017 11:59 PM UTC\n\nSecond Stage Start: 12/11/2017 11:59 PM UTC\n\nFinal Submission Deadline: 12/15/2017 11:59 PM UTC\n\n\nSo we have 3 dates now for model upload. And the rules pages shows the second stage starting after the date of the model upload/first stage deadline.\n\nCan you please clarify and make the doc pages consistent with the clarification.\n\nBrian.", "250475": "Hi Brian,\n\nThe model upload and first stage deadline is 12/10 - I've updated the rules to correct the typo. The Timeline tab has been accurate and supercedes.\n\nPlease read the FAQ page in the above post regarding how the two stage competitions work regarding model uploads and beginning/end of stages.\n\nThanks!", "250514": "I am good with the two stage. I thought the dates were odd on the rules page. Meant to ask before. Never noticed the timeline tab until now. In any event, thanks for clarifying.\n\nBrian.", "251334": "Just for the sake of clarity, you don't want us to actually upload the models themselves, you just want us to upload the code responsible for producing the models. Even though RNG means that executing the code again may produce slightly different results on subsequent runs?", "251731": "Per the FAQ: \"In order to achieve this, we ask that you upload your source code, including the correct parameters that you used for generating your submission files\" You can set the random seed.", "251779": "I reread the stage2 FAQ. To clarify your follow up statement: all we need to do is submit a program that when executed generates our submission csv file? The method of training or discovering such a program does not itself have to be submitted. In other words, we just need to submit end-to-end **inference code**, and not any training code?", "251786": "Training code is also required.", "251809": "For the purpose of confirming that our submission matches our model, will you be executing our inference program, or retraining everything from scratch and then running the inference program? When doing inference, I plan on loading model checkpoints to avoid retraining.\n\nMy main concern is that there are known non-deterministic ops in tensorflow that are used in calculating gradients (training): https://github.com/tensorflow/tensorflow/issues/2652, https://github.com/tensorflow/tensorflow/issues/2732, https://github.com/tensorflow/tensorflow/issues/3103\n\nI'm currently using tensorflow 1.3. As one of the comments pointed out, 1.4 might have solved this issue for some reductions but it's not guaranteed for all training ops. \n\nMy concern is that retraining might result in slightly different weights yielding slightly different predictions. However this is not an issue if you are only checking the final inference program.", "251937": "I'm just a contestant, but the impression I've gotten is that basically your submission will be reviewed \"in good faith.\" I'm sure they'll take your word for it and do their best to recreate your score; if necessary, they'll probably figure something out on-the-fly (in the case of a tie and utter failure in recreating a submission's predictions or something.)\n\nAfter reviewing the two-stage FAQ again (it's been a few weeks), I guess my point is that you should just upload the weights and largely bypass all of these concerns. I don't see any disadvantage to that, personally. Since the upload of models ends prior to the final test set being released, I doubt they'll be retraining models unless necessary. Other than pulling in \"altered\" weights under the guise of using a mostly pre-trained architecture, I don't see any reason to be suspicious of someone's uploaded model.\n\nAm I missing something?", "252043": "How should we submit permitted corrections for stage 2 (for example, changed paths to input data)? Should we submit weights for the model for both selected submissions of the stage 1?", "252059": "What is tested size limit for model upload? My model is about 50GB and I'm not sure if I can successfully upload it as single file using cable modem.", "252453": "By the nature of things, different random seeds, could, in principle, produce minor differences in the final result, so perhaps they'll rerun the model, if it doesn't produce a result consistent with your final prediction? And if they really can't reproduce it all, then there might be trouble, I imagine.", "252456": "Cable Modem? Is that a feature in gsutil?", "252457": "I use powerful personal computer for computation. I will use checksums and exclude optional data.", "252630": "I have literally zero experience with the kind of cloud computing platform they'd be using, but I'd wager Kaggle has access to development environments that would eat your powerful machine for breakfast. I don't think it's a concern, but I may be wrong :)", "252788": "Addison,\ncan you clarify the Dec4 deadline and confirm that models developed and trained entirely from Stage1 data need not be posted until Dec 10", "252846": "That is correct. The Dec 4 deadline is to post your pre-trained models and external data used to the \"official external data\" thread. You then have until Dec 10 to upload your model.", "253919": "I have a question.\n\nThroughout the competition, I've been improving my model using hyper-parameter optimization. I know that this is allowed by the rules as long as it's entirely automated, and it is. But this exploration is very time consuming and I've spent weeks or months training different models. So my question is, can I use the model I've already tuned (the same model I've used in stage 1) for stage 2 for prediction? Even though I can't retrain from scratch in the allotted time, my code and instructions would theoretically be able to produce the same model or something reasonably close to it. I would be submitting the model I've already trained so the exact same submission can be reproduced very quickly.\n\nTo simplify my question: Do I actually have to retrain everything from scratch, which might be impossible for me in the 5-4 days we have, or can I use the same model I used in stage 1, knowing that that model can be recreated from the code given more time?", "253925": "You can't use your scores even for automated optimization. If you don't use them, then your stage 1 model wouldn't need to be optimized again for the stage 2.", "253933": "I'm not using the validation set for the optimization, I'm using a split of the training set.", "253960": "Hi Kevin, \nMy understanding is that you do not need to retrain everything from scratch for stage 2, you can just use your existing model (trained weights) to predict directly on stage 2 once its released. In fact, I don't think you are allowed to change your model at all after its submitted for stage 1 but you would have the option to re-train (or fine-tune) with stage 1 test images/ground truth labels included during stage 2 since those labels will also be released for stage 2.", "254020": "I wonder why we would have to publish external data sources if it would be allowed to package unverifiable weights files.\n\nAlso, per Addison: \nPer the FAQ: \"In order to achieve this, we ask that you upload your source code, including the correct parameters that you used for generating your submission files\"", "254072": "I think the reason for posting external data is to not give any team an unfair **data** advantage. For example, if you're a data science company that has some proprietary extra data or models pre-trained on that extra data, it would be unfair to the other competitors unless that data was made publicly available to everyone. That's why you're required to post all your data/model sources.\n\nIn contrast, my weights came entirely from competition data and are verifiable, just not in the amount of time we have for stage 2, which is my main concern.", "255698": "Please provide official answer with clarification for the questions there: https://www.kaggle.com/c/passenger-screening-algorithm-challenge/discussion/45353", "255782": "why I can not see my uploaded model after I submitted? how can I confirm that my model has been submitted?", "255792": "You should be able to see your model under Team > Your Model > Download. If you don't see it there, you should probably try to re-upload it.", "255983": "And.. you have to reload the page, because the page automatically loaded doesn't know about your model,.... tsssk Kaggle, you got me scared for a minute!", "256008": "> **Wensu wrote**\n> \n> > why I can not see my uploaded model after I submitted? how can I confirm that my model has been submitted?\n\nThe same happened to me: my first upload was visible in the \"preview\", but disappeared within minutes.\n\nAnyway, this is my model's SHA256: f3e38d9eb1211593b2aefe842d69dce41f3d92f2295d7757fc8243b06878d61d in case this happens again.", "256012": "On second thought, it's definitely a refresh issue in the browser (I updated my model again, but it was still showing the old one in the preview)", "256017": "I had problems with seeing correct uploaded file as well. So here is my SHA256, just in case:\nb3944fff04223a7db38e08f1cdc54d8ddf78c55ca58b23ad399684b1c29b8dbc", "256026": "Same here. Not able to see uploaded models. here is my SHA256, just in case.\nfb2c531b20b5a4e6911f49cb120fbbaa6c2743b568dee8cf60468e134909dcfc", "256035": "I don't see where my model is uploaded either ¯\\_(ツ)_/¯\n\nSHA256: 24d30810ccccfc0fb042cf5bcd20aaa76418d8c2481dc6ab8d8b251bf5ea44a2", "256049": "It seemed to work just fine for me but now nothing shows anymore when I visit the \"team\" page. \nIs it normal to not see my submission listed anymore? (now without the \"Upload New File\" button of course).\nAs I also didn't get a receipt confirmation e-mail, just to make sure: the SHA256 for my submission file is: 841b48023fdaca68fab9aa480d3c4335f7478f2dcf86792c1848f017501080e1", "256051": "I also don't see my model anymore, dropping sha just to make sure: 64a88bb1d1e5ea290cfcdf2d481c637b089ce8fa194c52f97143a7128bff440b", "256060": "I'm pretty sure this is normal since it seems to have affected everybody. But I'll post my SHA256 too just in case...\nA6AE6D40CA3F5D3191A0BA03ABBA9042A20E8D62C6032DA465493E84ECCE26A7", "256404": "The sha 256 hash for team TomaDon submission file is:\n78d5b48adaed987ea954fe43a15bbc0f130df8768f242125deaea6b5a74ead0a", "256542": "to be on the safe side. 2832290d32b72074018fc21485731395f4a1d00778466ae498476fc1df7f21ca model_submission_1512486248.tar.gz" }, "source": "meta" }