We want to start by saying Thank You to all Kagglers and teams for a great competition! We're very happy with the results so far and are looking forward to meeting the winning teams.
\n\nThe main competition may be over, but there is still a prize left to compete for. Now it's time for the special prize evaluation!
\n\nBelow we describe the evaluation process including what teams will be eligible to participate.
\n\nUpdate 2019-01-21
\n\nSubmission hardware limits update:\nThe test server won't have an integrated intel gpu.
\n\nUpdate 2019-01-15
\n\nWe've considered the feedback about participation and eligible teams and decided to update and clarify the rules for participation and tie breaking in the special prize evaluation.
\n\nWho
\n\nTop 1000 teams (1-1000) in the final private leaderboard in the main competition, may submit their models.
\n\nTimeline
\n\nHow
\n\nYOUR_TEAM_NAME_F1_SCORE.zip where YOUR_TEAM_NAME is the name of your team, and F1_SCORE is the submitted score in the main competition that we should compare to, when evaluating the submission. This must be one of two selected submissions in the main competition.\nAdditional instructions\nIt's important that the instructions are followed carefully, especially for the docker file and the Python submission template module that we will use to time the prediction speed of the private test set images. If we can't run the docker container or if the submission template module isn't compatible with our evaluation module, we can't guarantee that we will have time to evaluate the submission. To save time we won't make any modifications to submitted modules to make them runnable.
\n\nThis also means that we expect a default path to the file or directory that stores the saved model in the Model class. The default path should be set so that we can instantiate the Model class without specifying a path to the saved model, and the class should make sure that the model is properly loaded and set up. Ie it should be enough to do the following to create a ready model instance:
\n\n\nmodel = Model()\n
Then we should be able to call the predict method and run the prediction:
\nprediction = model.predict(pixel_data)\n
We will build the docker image from the docker file, run the docker container and bind mount our evaluation module. The evaluation module will import the Model class from the submission template module submission_predict.py that should be located in the working directory root of the container and run the prediction using the instantiated model. We expect a Python 3 environment. But we won't disqualify submissions that complete successfully in a Python 2 environment.
Sponsor evaluation steps outline
\n\nTo ensure that submitting teams can test the compatibility of their submissions before submitting, we have included a mock evaluation module special_eval.py and example docker file Dockerfile below. Using those together with the submission template module, that has the Model class, it should be possible to validate that the submission is correct. We also include an updated example submission template module submission_predict.py with example how to set a default path to a saved model.
Test compatibility
\n\nBuild the example docker image from the included example docker file from the root of the directory that has the docker file and the rest of the model files. This may require root access (sudo).
\n\n\ndocker build -t NAME_OF_EXAMPLE_DOCKER_IMAGE .\n
Run the example container based on the included example docker file. This may require root access (sudo).
\n\n\ndocker run --rm \\\n-v /PATH/TO/special_eval.py:/app/special_eval.py \\\nNAME_OF_EXAMPLE_DOCKER_IMAGE python special_eval.py\n
This should print a simple Python dictionary representation of the result. Only two mock test samples are included. Eg:
\n\n\n{'1': ({'0'}, set(), 8.58306884765625e-06), '2': ({'25', '0'}, set(), 9.298324584960938e-06)}\n
Good speed!
", "messages": [ { "id": "455014", "postDate": "01/12/2019 18:45:30", "content": "We want to start by saying Thank You to all Kagglers and teams for a great competition! We're very happy with the results so far and are looking forward to meeting the winning teams.
\n\nThe main competition may be over, but there is still a prize left to compete for. Now it's time for the special prize evaluation!
\n\nBelow we describe the evaluation process including what teams will be eligible to participate.
\n\nUpdate 2019-01-21
\n\nSubmission hardware limits update:\nThe test server won't have an integrated intel gpu.
\n\nUpdate 2019-01-15
\n\nWe've considered the feedback about participation and eligible teams and decided to update and clarify the rules for participation and tie breaking in the special prize evaluation.
\n\nWho
\n\nTop 1000 teams (1-1000) in the final private leaderboard in the main competition, may submit their models.
\n\nTimeline
\n\nHow
\n\nYOUR_TEAM_NAME_F1_SCORE.zip where YOUR_TEAM_NAME is the name of your team, and F1_SCORE is the submitted score in the main competition that we should compare to, when evaluating the submission. This must be one of two selected submissions in the main competition.\nAdditional instructions\nIt's important that the instructions are followed carefully, especially for the docker file and the Python submission template module that we will use to time the prediction speed of the private test set images. If we can't run the docker container or if the submission template module isn't compatible with our evaluation module, we can't guarantee that we will have time to evaluate the submission. To save time we won't make any modifications to submitted modules to make them runnable.
\n\nThis also means that we expect a default path to the file or directory that stores the saved model in the Model class. The default path should be set so that we can instantiate the Model class without specifying a path to the saved model, and the class should make sure that the model is properly loaded and set up. Ie it should be enough to do the following to create a ready model instance:
\n\n\nmodel = Model()\n
Then we should be able to call the predict method and run the prediction:
\nprediction = model.predict(pixel_data)\n
We will build the docker image from the docker file, run the docker container and bind mount our evaluation module. The evaluation module will import the Model class from the submission template module submission_predict.py that should be located in the working directory root of the container and run the prediction using the instantiated model. We expect a Python 3 environment. But we won't disqualify submissions that complete successfully in a Python 2 environment.
Sponsor evaluation steps outline
\n\nTo ensure that submitting teams can test the compatibility of their submissions before submitting, we have included a mock evaluation module special_eval.py and example docker file Dockerfile below. Using those together with the submission template module, that has the Model class, it should be possible to validate that the submission is correct. We also include an updated example submission template module submission_predict.py with example how to set a default path to a saved model.
Test compatibility
\n\nBuild the example docker image from the included example docker file from the root of the directory that has the docker file and the rest of the model files. This may require root access (sudo).
\n\n\ndocker build -t NAME_OF_EXAMPLE_DOCKER_IMAGE .\n
Run the example container based on the included example docker file. This may require root access (sudo).
\n\n\ndocker run --rm \\\n-v /PATH/TO/special_eval.py:/app/special_eval.py \\\nNAME_OF_EXAMPLE_DOCKER_IMAGE python special_eval.py\n
This should print a simple Python dictionary representation of the result. Only two mock test samples are included. Eg:
\n\n\n{'1': ({'0'}, set(), 8.58306884765625e-06), '2': ({'25', '0'}, set(), 9.298324584960938e-06)}\n
Good speed!
", "rawMarkdown": "We want to start by saying Thank You to all Kagglers and teams for a great competition! We're very happy with the results so far and are looking forward to meeting the winning teams.\n\nThe main competition may be over, but there is still a prize left to compete for. Now it's time for the special prize evaluation!\n\nBelow we describe the evaluation process including what teams will be eligible to participate.\n\n**Update 2019-01-21**\n\nSubmission hardware limits update:\nThe test server won't have an integrated intel gpu.\n\n**Update 2019-01-15**\n\nWe've considered the feedback about participation and eligible teams and decided to update and clarify the rules for participation and tie breaking in the special prize evaluation.\n\n- The top 1000 (1-1000) teams will now be eligible to participate in the special prize evaluation.\n- Beyond submitting the zip-file, a participating team must also fill in our [survey](https://docs.google.com/forms/d/e/1FAIpQLScFRTpX57XyW0nn0K1NVfk8oCQEMd9wD_9ALwlMD3yN4cDoVA/viewform?usp=sf_link%20%27survey%27) about the main competition. The survey deadline (January 16, 2019) as posted in the survey discussion [thread](https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/77367) is only applied for teams that want to participate as co-authors in the paper. For the special prize evaluation the survey can be filled in until January 25, 2019.\n- Where more than one team completes the evaluation with the same time, the team with the higher macro F1 score in the special prize evaluation will be preferred and win the tie. It doesn't matter how close a team's F1 score in the special prize evaluation is to the team's F1 score in the main competition, as long as it's within +/- 1 absolute percentage point.\n\n**Who**\n\nTop 1000 teams (1-1000) in the final private leaderboard in the main competition, may submit their models.\n\n**Timeline**\n\n- January 25, 2019 - Deadline to submit for special prize, for participating teams.\n- February 15, 2019 - Deadline for evaluation of special prize, for Sponsor.\n\n**How**\n\n- Fill in our [survey](https://docs.google.com/forms/d/e/1FAIpQLScFRTpX57XyW0nn0K1NVfk8oCQEMd9wD_9ALwlMD3yN4cDoVA/viewform?usp=sf_link%20%27survey%27) about the main competition. \n- Upload a zip file to the team [tab](https://www.kaggle.com/c/human-protein-atlas-image-classification/team) according to previously posted [instructions](https://www.kaggle.com/c/human-protein-atlas-image-classification#Special-Prize-Instructions) and additional instructions below.\n- Name the zip file `YOUR_TEAM_NAME_F1_SCORE.zip` where `YOUR_TEAM_NAME` is the name of your team, and `F1_SCORE` is the submitted score in the main competition that we should compare to, when evaluating the submission. This must be one of two selected submissions in the main competition.\n - Remember that the special prize submitted model must maintain its score from the main competition +/- 1 absolute percentage point. Eg a score of 0.50 in the main competition will have an acceptable interval of 0.49 - 0.51 in the special prize evaluation.\n\n**Additional instructions**\nIt's important that the instructions are followed carefully, especially for the docker file and the Python submission template module that we will use to time the prediction speed of the private test set images. If we can't run the docker container or if the submission template module isn't compatible with our evaluation module, we can't guarantee that we will have time to evaluate the submission. To save time **we won't make any modifications to submitted modules to make them runnable**.\n\nThis also means that we expect a default path to the file or directory that stores the saved model in the Model class. The default path should be set so that we can instantiate the Model class without specifying a path to the saved model, and the class should make sure that the model is properly loaded and set up. Ie it should be enough to do the following to create a ready model instance:\n\n```\nmodel = Model()\n```\n\nThen we should be able to call the `predict` method and run the prediction:\n\n```\nprediction = model.predict(pixel_data)\n```\n\nWe will build the docker image from the docker file, run the docker container and bind mount our evaluation module. The evaluation module will import the Model class from **the submission template module `submission_predict.py` that should be located in the working directory root of the container** and run the prediction using the instantiated model. **We expect a Python 3 environment.** But we won't disqualify submissions that complete successfully in a Python 2 environment.\n\n---\n\n*Sponsor evaluation steps outline*\n\n1. Build docker image from docker file.\n2. Run docker container from docker image.\n - Bind mount our Sponsor evaluation module into the container.\n3. Import Model class from submission template module.\n4. Load test data.\n5. Instantiate model from Model class.\n6. Iterate over test data.\n - Load one field of view images.\n - Time start.\n - Predict one field of view images data using the instantiated model.\n - Time end.\n - Save results in a dictionary.\n7. Return results.\n8. Sum prediction time for all test images.\n\n---\n\nTo ensure that submitting teams can test the compatibility of their submissions before submitting, we have included a mock evaluation module `special_eval.py` and example docker file `Dockerfile` below. Using those together with the submission template module, that has the Model class, it should be possible to validate that the submission is correct. We also include an updated example submission template module `submission_predict.py` with example how to set a default path to a saved model.\n\n**Test compatibility**\n\n- Build the example docker image from the included example docker file from the root of the directory that has the docker file and the rest of the model files. This may require root access (sudo).\n\n ```\n docker build -t NAME_OF_EXAMPLE_DOCKER_IMAGE .\n ```\n\n- Run the example container based on the included example docker file. This may require root access (sudo).\n\n ```\n docker run --rm \\\n -v /PATH/TO/special_eval.py:/app/special_eval.py \\\n NAME_OF_EXAMPLE_DOCKER_IMAGE python special_eval.py\n ```\n\nThis should print a simple Python dictionary representation of the result. Only two mock test samples are included. Eg:\n\n```\n{'1': ({'0'}, set(), 8.58306884765625e-06), '2': ({'25', '0'}, set(), 9.298324584960938e-06)}\n```\n\nGood speed!", "votes": null }, { "id": "455041", "postDate": "01/12/2019 20:06:24", "content": "I would like to comment (like several others did), that you might want to reconsider the special prize participation rules. There is a huge difference between designing a winning model for the competition (14 models ensembled with 8 tta...) and having the best slim and fast model for the special prize. Having only participants (and, more importantly, solutions) from the top 200 compete in this second part will greatly reduce your exposure to solutions that actually match the special prize requirements. It would be interesting if you could actually open the docker submission server running for all competitors of this competition, maybe not for the special prize, but just for fun.
", "rawMarkdown": "I would like to comment (like several others did), that you might want to reconsider the special prize participation rules. There is a huge difference between designing a winning model for the competition (14 models ensembled with 8 tta...) and having the best slim and fast model for the special prize. Having only participants (and, more importantly, solutions) from the top 200 compete in this second part will greatly reduce your exposure to solutions that actually match the special prize requirements. It would be interesting if you could actually open the docker submission server running for all competitors of this competition, maybe not for the special prize, but just for fun.", "votes": null }, { "id": "455064", "postDate": "01/12/2019 21:27:05", "content": "\n\n\nRemember that the special prize submitted model must maintain its score from the main competition +/- 1 absolute percentage point. Eg a score of 0.50 in the main competition will have an acceptable interval of 0.49 - 0.51 in the special prize evaluation.
\n
Is this the private leaderboard or the public one?
", "rawMarkdown": "> Remember that the special prize submitted model must maintain its score from the main competition +/- 1 absolute percentage point. Eg a score of 0.50 in the main competition will have an acceptable interval of 0.49 - 0.51 in the special prize evaluation.\n\nIs this the private leaderboard or the public one?", "votes": null }, { "id": "455106", "postDate": "01/13/2019 01:42:21", "content": "If I fit a simpler model using only the test data and the target generated by my complex model, will you accept?\nThe model will be totally useless in practice but seem not violate the rules.
", "rawMarkdown": "If I fit a simpler model using only the test data and the target generated by my complex model, will you accept?\nThe model will be totally useless in practice but seem not violate the rules.", "votes": null }, { "id": "455573", "postDate": "01/14/2019 07:13:46", "content": "Since the 14 models ensemble most likely references our solution, I have to notice that our best single model gets top-20 on the private leaderboard ;)
\n\nBut unfortunately it is not eligible for the special prize contest because it wasn't amongst two selected submissions.
", "rawMarkdown": "Since the 14 models ensemble most likely references our solution, I have to notice that our best single model gets top-20 on the private leaderboard ;)\n\nBut unfortunately it is not eligible for the special prize contest because it wasn't amongst two selected submissions.", "votes": null }, { "id": "455680", "postDate": "01/14/2019 11:02:40", "content": "\n\n\nBut unfortunately it is not eligible for the special prize contest because it wasn't amongst two selected submissions.
\n
Same for us. Additionally I don't see the value added for the organizers. If they want to implement a fast and accurate model in production, in my opinion it would have been better to allow for model compression as done in speech recognition challenge, and one could simply compress the top ensembles into one fast model. And even if model compression would be allowed, the rule of +/- 1 absolute percentage point seems odd. With same speed it would prefer a 0.51 -> 0.50 model over a 0.56 -> 0.54 model
", "rawMarkdown": "> But unfortunately it is not eligible for the special prize contest because it wasn't amongst two selected submissions.\n\nSame for us. Additionally I don't see the value added for the organizers. If they want to implement a fast and accurate model in production, in my opinion it would have been better to allow for model compression as done in speech recognition challenge, and one could simply compress the top ensembles into one fast model. And even if model compression would be allowed, the rule of +/- 1 absolute percentage point seems odd. With same speed it would prefer a 0.51 -> 0.50 model over a 0.56 -> 0.54 model", "votes": null }, { "id": "455880", "postDate": "01/14/2019 19:02:14", "content": "It was a memorable ensemble ;)
", "rawMarkdown": "It was a memorable ensemble ;)", "votes": null }, { "id": "456263", "postDate": "01/15/2019 12:51:22", "content": "We've considered the feedback about participation and eligible teams and decided to update and clarify the rules for participation and tie breaking in the special prize evaluation.
\n\nNo, the same model must be used as submitted in the main competition.
", "rawMarkdown": "No, the same model must be used as submitted in the main competition.", "votes": null }, { "id": "456269", "postDate": "01/15/2019 12:59:26", "content": "It's the private leaderboard.
", "rawMarkdown": "It's the private leaderboard.", "votes": null }, { "id": "456431", "postDate": "01/15/2019 20:00:58", "content": "Thank you for considering the feedback. Again, i would like to bring to your attention the fact that this is actually against your best interests. Teams that submitted 14 models ensamble with 8 tta (:)) might have a really good single model that might fit the docker environment but it won't be within 1 percentage point. Why not just remove this condition, put some absolute barrier on the f1 (derived from your current solution) and let us do our bestest?
", "rawMarkdown": "Thank you for considering the feedback. Again, i would like to bring to your attention the fact that this is actually against your best interests. Teams that submitted 14 models ensamble with 8 tta (:)) might have a really good single model that might fit the docker environment but it won't be within 1 percentage point. Why not just remove this condition, put some absolute barrier on the f1 (derived from your current solution) and let us do our bestest?", "votes": null }, { "id": "456780", "postDate": "01/16/2019 14:41:40", "content": "Since it was possible to mark two submissins for evaluation, a team could submit two different approaches, eg ensemble and singel model, in the main competition.
", "rawMarkdown": "Since it was possible to mark two submissins for evaluation, a team could submit two different approaches, eg ensemble and singel model, in the main competition.", "votes": null }, { "id": "456810", "postDate": "01/16/2019 15:23:59", "content": "Since it was possible to mark two submissins for evaluation, a team could submit two different approaches, eg ensemble and singel model, in the main competition.\n\n
Yes, teams could do it, but nobody actually did it. It is useless to select single model in the main competition since ensemble is more stable and usually more accurate.
", "rawMarkdown": "Since it was possible to mark two submissins for evaluation, a team could submit two different approaches, eg ensemble and singel model, in the main competition.\nYes, teams **could** do it, but nobody actually did it. It is useless to select single model in the main competition since ensemble is more stable and usually more accurate.", "votes": null }, { "id": "456820", "postDate": "01/16/2019 15:48:29", "content": "
\n\n\nbut nobody actually did it
\n
apart from @bestfitting :D
", "rawMarkdown": "> but nobody actually did it\n\napart from @bestfitting :D", "votes": null }, { "id": "456823", "postDate": "01/16/2019 16:00:18", "content": "Yeah, probably :)
", "rawMarkdown": "Yeah, probably :)", "votes": null }, { "id": "458694", "postDate": "01/20/2019 09:37:29", "content": "@martinhjelmare, will the docker test server have the integrated intel gpu? what kind?
", "rawMarkdown": "martinhjelmare, will the docker test server have the integrated intel gpu? what kind?", "votes": null }, { "id": "458696", "postDate": "01/20/2019 09:43:00", "content": "And yet again, I would like to comment how strange the requirement of being within +-1 of the f1 score of the competition is not realistic. Many teams (mine included) used a variable th based on class distribution. This mean that after we predict the whole test dataset, we select a th that will allow the relative number of entries from this class. Naturally, this can't be done prediction by prediction.
", "rawMarkdown": "And yet again, I would like to comment how strange the requirement of being within +-1 of the f1 score of the competition is not realistic. Many teams (mine included) used a variable th based on class distribution. This mean that after we predict the whole test dataset, we select a th that will allow the relative number of entries from this class. Naturally, this can't be done prediction by prediction.", "votes": null }, { "id": "458949", "postDate": "01/20/2019 21:47:34", "content": "also, the evaluation matrix is a tad unclear here. is speed part of it if its under 1h? or is it f1 like the competition? and what about leak data?
", "rawMarkdown": "also, the evaluation matrix is a tad unclear here. is speed part of it if its under 1h? or is it f1 like the competition? and what about leak data?", "votes": null }, { "id": "459178", "postDate": "01/21/2019 10:55:18", "content": "just to make it easier for us, can you share the exact parameter you are going to run the docker image with? probably with 4096mb memory and 2 cores, but would be nice to know exactly how its going to be run.
", "rawMarkdown": "just to make it easier for us, can you share the exact parameter you are going to run the docker image with? probably with 4096mb memory and 2 cores, but would be nice to know exactly how its going to be run.", "votes": null }, { "id": "459360", "postDate": "01/21/2019 16:07:44", "content": "No. The test server won't have an integrated intel gpu. To allow more teams to participate we have to scale up to a server, and simplify hardware environment requirements.
", "rawMarkdown": "No. The test server won't have an integrated intel gpu. To allow more teams to participate we have to scale up to a server, and simplify hardware environment requirements.", "votes": null }, { "id": "459364", "postDate": "01/21/2019 16:09:59", "content": "Speed (time) is the primary evaluation criteria. If there's a tie, higher f1 score wins.
", "rawMarkdown": "Speed (time) is the primary evaluation criteria. If there's a tie, higher f1 score wins.", "votes": null }, { "id": "459372", "postDate": "01/21/2019 16:18:15", "content": "We can't share exact run parameters at this time. We're aiming to run the evaluation on a node with 2 cores and 4096 MB memory.
", "rawMarkdown": "We can't share exact run parameters at this time. We're aiming to run the evaluation on a node with 2 cores and 4096 MB memory.", "votes": null }, { "id": "459541", "postDate": "01/22/2019 00:24:02", "content": "Is using leak data allowed? It doesn't make any sense but it was used in the competition...
", "rawMarkdown": "Is using leak data allowed? It doesn't make any sense but it was used in the competition...", "votes": null }, { "id": "460555", "postDate": "01/24/2019 00:23:48", "content": "And one more question.... 25th of Jan what time zone lol
", "rawMarkdown": "And one more question.... 25th of Jan what time zone lol", "votes": null }, { "id": "460957", "postDate": "01/24/2019 21:19:31", "content": "The deadline for submission will be Jan 25, 2019 at 11:59 p.m. PST.
", "rawMarkdown": "The deadline for submission will be Jan 25, 2019 at 11:59 p.m. PST.", "votes": null }, { "id": "470665", "postDate": "02/13/2019 10:48:49", "content": "We are now finished with evaluation and are happy to announce the winning team:
\n\nCongratulations to team Protein Shake! Their average prediction time per FOV was 67 ms.
", "rawMarkdown": "We are now finished with evaluation and are happy to announce the winning team:\n\nCongratulations to team Protein Shake! Their average prediction time per FOV was 67 ms.", "votes": null }, { "id": "470929", "postDate": "02/13/2019 19:25:43", "content": "Whooohooo!!!! This is great!!!!\nThanks for doing this real life special prize. I enjoyed tackling the constraints very much.
", "rawMarkdown": "Whooohooo!!!! This is great!!!!\nThanks for doing this real life special prize. I enjoyed tackling the constraints very much.", "votes": null } ], "comments": [ { "id": 455041, "author_name": "moshel", "author_url": "", "post_date": "01/12/2019 20:06:24", "content": "I would like to comment (like several others did), that you might want to reconsider the special prize participation rules. There is a huge difference between designing a winning model for the competition (14 models ensembled with 8 tta...) and having the best slim and fast model for the special prize. Having only participants (and, more importantly, solutions) from the top 200 compete in this second part will greatly reduce your exposure to solutions that actually match the special prize requirements. It would be interesting if you could actually open the docker submission server running for all competitors of this competition, maybe not for the special prize, but just for fun.
", "votes": null, "replies": [ { "id": 455573, "author_name": "hokmund", "author_url": "", "post_date": "01/14/2019 07:13:46", "content": "Since the 14 models ensemble most likely references our solution, I have to notice that our best single model gets top-20 on the private leaderboard ;)
\n\nBut unfortunately it is not eligible for the special prize contest because it wasn't amongst two selected submissions.
", "votes": null, "replies": [] }, { "id": 455880, "author_name": "moshel", "author_url": "", "post_date": "01/14/2019 19:02:14", "content": "It was a memorable ensemble ;)
", "votes": null, "replies": [] } ] }, { "id": 455064, "author_name": "alexanderliao", "author_url": "", "post_date": "01/12/2019 21:27:05", "content": "\n\n\nRemember that the special prize submitted model must maintain its score from the main competition +/- 1 absolute percentage point. Eg a score of 0.50 in the main competition will have an acceptable interval of 0.49 - 0.51 in the special prize evaluation.
\n
Is this the private leaderboard or the public one?
", "votes": null, "replies": [ { "id": 456269, "author_name": "martinhjelmare", "author_url": "", "post_date": "01/15/2019 12:59:26", "content": "It's the private leaderboard.
", "votes": null, "replies": [] } ] }, { "id": 455106, "author_name": "wowfattie", "author_url": "", "post_date": "01/13/2019 01:42:21", "content": "If I fit a simpler model using only the test data and the target generated by my complex model, will you accept?\nThe model will be totally useless in practice but seem not violate the rules.
", "votes": null, "replies": [ { "id": 456268, "author_name": "martinhjelmare", "author_url": "", "post_date": "01/15/2019 12:59:04", "content": "No, the same model must be used as submitted in the main competition.
", "votes": null, "replies": [] } ] }, { "id": 455680, "author_name": "christofhenkel", "author_url": "", "post_date": "01/14/2019 11:02:40", "content": "\n\n\nBut unfortunately it is not eligible for the special prize contest because it wasn't amongst two selected submissions.
\n
Same for us. Additionally I don't see the value added for the organizers. If they want to implement a fast and accurate model in production, in my opinion it would have been better to allow for model compression as done in speech recognition challenge, and one could simply compress the top ensembles into one fast model. And even if model compression would be allowed, the rule of +/- 1 absolute percentage point seems odd. With same speed it would prefer a 0.51 -> 0.50 model over a 0.56 -> 0.54 model
", "votes": null, "replies": [] }, { "id": 456263, "author_name": "martinhjelmare", "author_url": "", "post_date": "01/15/2019 12:51:22", "content": "We've considered the feedback about participation and eligible teams and decided to update and clarify the rules for participation and tie breaking in the special prize evaluation.
\n\nThank you for considering the feedback. Again, i would like to bring to your attention the fact that this is actually against your best interests. Teams that submitted 14 models ensamble with 8 tta (:)) might have a really good single model that might fit the docker environment but it won't be within 1 percentage point. Why not just remove this condition, put some absolute barrier on the f1 (derived from your current solution) and let us do our bestest?
", "votes": null, "replies": [] }, { "id": 456780, "author_name": "martinhjelmare", "author_url": "", "post_date": "01/16/2019 14:41:40", "content": "Since it was possible to mark two submissins for evaluation, a team could submit two different approaches, eg ensemble and singel model, in the main competition.
", "votes": null, "replies": [] }, { "id": 456810, "author_name": "hokmund", "author_url": "", "post_date": "01/16/2019 15:23:59", "content": "Since it was possible to mark two submissins for evaluation, a team could submit two different approaches, eg ensemble and singel model, in the main competition.\n\n
Yes, teams could do it, but nobody actually did it. It is useless to select single model in the main competition since ensemble is more stable and usually more accurate.
", "votes": null, "replies": [] }, { "id": 456820, "author_name": "christofhenkel", "author_url": "", "post_date": "01/16/2019 15:48:29", "content": "\n\n\nbut nobody actually did it
\n
apart from @bestfitting :D
", "votes": null, "replies": [] }, { "id": 456823, "author_name": "hokmund", "author_url": "", "post_date": "01/16/2019 16:00:18", "content": "Yeah, probably :)
", "votes": null, "replies": [] } ] }, { "id": 458694, "author_name": "moshel", "author_url": "", "post_date": "01/20/2019 09:37:29", "content": "@martinhjelmare, will the docker test server have the integrated intel gpu? what kind?
", "votes": null, "replies": [ { "id": 459360, "author_name": "martinhjelmare", "author_url": "", "post_date": "01/21/2019 16:07:44", "content": "No. The test server won't have an integrated intel gpu. To allow more teams to participate we have to scale up to a server, and simplify hardware environment requirements.
", "votes": null, "replies": [] } ] }, { "id": 458696, "author_name": "moshel", "author_url": "", "post_date": "01/20/2019 09:43:00", "content": "And yet again, I would like to comment how strange the requirement of being within +-1 of the f1 score of the competition is not realistic. Many teams (mine included) used a variable th based on class distribution. This mean that after we predict the whole test dataset, we select a th that will allow the relative number of entries from this class. Naturally, this can't be done prediction by prediction.
", "votes": null, "replies": [] }, { "id": 458949, "author_name": "moshel", "author_url": "", "post_date": "01/20/2019 21:47:34", "content": "also, the evaluation matrix is a tad unclear here. is speed part of it if its under 1h? or is it f1 like the competition? and what about leak data?
", "votes": null, "replies": [ { "id": 459364, "author_name": "martinhjelmare", "author_url": "", "post_date": "01/21/2019 16:09:59", "content": "Speed (time) is the primary evaluation criteria. If there's a tie, higher f1 score wins.
", "votes": null, "replies": [] }, { "id": 459541, "author_name": "moshel", "author_url": "", "post_date": "01/22/2019 00:24:02", "content": "Is using leak data allowed? It doesn't make any sense but it was used in the competition...
", "votes": null, "replies": [] } ] }, { "id": 459178, "author_name": "moshel", "author_url": "", "post_date": "01/21/2019 10:55:18", "content": "just to make it easier for us, can you share the exact parameter you are going to run the docker image with? probably with 4096mb memory and 2 cores, but would be nice to know exactly how its going to be run.
", "votes": null, "replies": [ { "id": 459372, "author_name": "martinhjelmare", "author_url": "", "post_date": "01/21/2019 16:18:15", "content": "We can't share exact run parameters at this time. We're aiming to run the evaluation on a node with 2 cores and 4096 MB memory.
", "votes": null, "replies": [] } ] }, { "id": 460555, "author_name": "moshel", "author_url": "", "post_date": "01/24/2019 00:23:48", "content": "And one more question.... 25th of Jan what time zone lol
", "votes": null, "replies": [ { "id": 460957, "author_name": "elizabethpark", "author_url": "", "post_date": "01/24/2019 21:19:31", "content": "The deadline for submission will be Jan 25, 2019 at 11:59 p.m. PST.
", "votes": null, "replies": [] } ] }, { "id": 470665, "author_name": "martinhjelmare", "author_url": "", "post_date": "02/13/2019 10:48:49", "content": "We are now finished with evaluation and are happy to announce the winning team:
\n\nCongratulations to team Protein Shake! Their average prediction time per FOV was 67 ms.
", "votes": null, "replies": [ { "id": 470929, "author_name": "moshel", "author_url": "", "post_date": "02/13/2019 19:25:43", "content": "Whooohooo!!!! This is great!!!!\nThanks for doing this real life special prize. I enjoyed tackling the constraints very much.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "455014": "We want to start by saying Thank You to all Kagglers and teams for a great competition! We're very happy with the results so far and are looking forward to meeting the winning teams.\n\nThe main competition may be over, but there is still a prize left to compete for. Now it's time for the special prize evaluation!\n\nBelow we describe the evaluation process including what teams will be eligible to participate.\n\n**Update 2019-01-21**\n\nSubmission hardware limits update:\nThe test server won't have an integrated intel gpu.\n\n**Update 2019-01-15**\n\nWe've considered the feedback about participation and eligible teams and decided to update and clarify the rules for participation and tie breaking in the special prize evaluation.\n\n- The top 1000 (1-1000) teams will now be eligible to participate in the special prize evaluation.\n- Beyond submitting the zip-file, a participating team must also fill in our [survey](https://docs.google.com/forms/d/e/1FAIpQLScFRTpX57XyW0nn0K1NVfk8oCQEMd9wD_9ALwlMD3yN4cDoVA/viewform?usp=sf_link%20%27survey%27) about the main competition. The survey deadline (January 16, 2019) as posted in the survey discussion [thread](https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/77367) is only applied for teams that want to participate as co-authors in the paper. For the special prize evaluation the survey can be filled in until January 25, 2019.\n- Where more than one team completes the evaluation with the same time, the team with the higher macro F1 score in the special prize evaluation will be preferred and win the tie. It doesn't matter how close a team's F1 score in the special prize evaluation is to the team's F1 score in the main competition, as long as it's within +/- 1 absolute percentage point.\n\n**Who**\n\nTop 1000 teams (1-1000) in the final private leaderboard in the main competition, may submit their models.\n\n**Timeline**\n\n- January 25, 2019 - Deadline to submit for special prize, for participating teams.\n- February 15, 2019 - Deadline for evaluation of special prize, for Sponsor.\n\n**How**\n\n- Fill in our [survey](https://docs.google.com/forms/d/e/1FAIpQLScFRTpX57XyW0nn0K1NVfk8oCQEMd9wD_9ALwlMD3yN4cDoVA/viewform?usp=sf_link%20%27survey%27) about the main competition. \n- Upload a zip file to the team [tab](https://www.kaggle.com/c/human-protein-atlas-image-classification/team) according to previously posted [instructions](https://www.kaggle.com/c/human-protein-atlas-image-classification#Special-Prize-Instructions) and additional instructions below.\n- Name the zip file `YOUR_TEAM_NAME_F1_SCORE.zip` where `YOUR_TEAM_NAME` is the name of your team, and `F1_SCORE` is the submitted score in the main competition that we should compare to, when evaluating the submission. This must be one of two selected submissions in the main competition.\n - Remember that the special prize submitted model must maintain its score from the main competition +/- 1 absolute percentage point. Eg a score of 0.50 in the main competition will have an acceptable interval of 0.49 - 0.51 in the special prize evaluation.\n\n**Additional instructions**\nIt's important that the instructions are followed carefully, especially for the docker file and the Python submission template module that we will use to time the prediction speed of the private test set images. If we can't run the docker container or if the submission template module isn't compatible with our evaluation module, we can't guarantee that we will have time to evaluate the submission. To save time **we won't make any modifications to submitted modules to make them runnable**.\n\nThis also means that we expect a default path to the file or directory that stores the saved model in the Model class. The default path should be set so that we can instantiate the Model class without specifying a path to the saved model, and the class should make sure that the model is properly loaded and set up. Ie it should be enough to do the following to create a ready model instance:\n\n```\nmodel = Model()\n```\n\nThen we should be able to call the `predict` method and run the prediction:\n\n```\nprediction = model.predict(pixel_data)\n```\n\nWe will build the docker image from the docker file, run the docker container and bind mount our evaluation module. The evaluation module will import the Model class from **the submission template module `submission_predict.py` that should be located in the working directory root of the container** and run the prediction using the instantiated model. **We expect a Python 3 environment.** But we won't disqualify submissions that complete successfully in a Python 2 environment.\n\n---\n\n*Sponsor evaluation steps outline*\n\n1. Build docker image from docker file.\n2. Run docker container from docker image.\n - Bind mount our Sponsor evaluation module into the container.\n3. Import Model class from submission template module.\n4. Load test data.\n5. Instantiate model from Model class.\n6. Iterate over test data.\n - Load one field of view images.\n - Time start.\n - Predict one field of view images data using the instantiated model.\n - Time end.\n - Save results in a dictionary.\n7. Return results.\n8. Sum prediction time for all test images.\n\n---\n\nTo ensure that submitting teams can test the compatibility of their submissions before submitting, we have included a mock evaluation module `special_eval.py` and example docker file `Dockerfile` below. Using those together with the submission template module, that has the Model class, it should be possible to validate that the submission is correct. We also include an updated example submission template module `submission_predict.py` with example how to set a default path to a saved model.\n\n**Test compatibility**\n\n- Build the example docker image from the included example docker file from the root of the directory that has the docker file and the rest of the model files. This may require root access (sudo).\n\n ```\n docker build -t NAME_OF_EXAMPLE_DOCKER_IMAGE .\n ```\n\n- Run the example container based on the included example docker file. This may require root access (sudo).\n\n ```\n docker run --rm \\\n -v /PATH/TO/special_eval.py:/app/special_eval.py \\\n NAME_OF_EXAMPLE_DOCKER_IMAGE python special_eval.py\n ```\n\nThis should print a simple Python dictionary representation of the result. Only two mock test samples are included. Eg:\n\n```\n{'1': ({'0'}, set(), 8.58306884765625e-06), '2': ({'25', '0'}, set(), 9.298324584960938e-06)}\n```\n\nGood speed!", "455041": "I would like to comment (like several others did), that you might want to reconsider the special prize participation rules. There is a huge difference between designing a winning model for the competition (14 models ensembled with 8 tta...) and having the best slim and fast model for the special prize. Having only participants (and, more importantly, solutions) from the top 200 compete in this second part will greatly reduce your exposure to solutions that actually match the special prize requirements. It would be interesting if you could actually open the docker submission server running for all competitors of this competition, maybe not for the special prize, but just for fun.", "455064": "> Remember that the special prize submitted model must maintain its score from the main competition +/- 1 absolute percentage point. Eg a score of 0.50 in the main competition will have an acceptable interval of 0.49 - 0.51 in the special prize evaluation.\n\nIs this the private leaderboard or the public one?", "455106": "If I fit a simpler model using only the test data and the target generated by my complex model, will you accept?\nThe model will be totally useless in practice but seem not violate the rules.", "455573": "Since the 14 models ensemble most likely references our solution, I have to notice that our best single model gets top-20 on the private leaderboard ;)\n\nBut unfortunately it is not eligible for the special prize contest because it wasn't amongst two selected submissions.", "455680": "> But unfortunately it is not eligible for the special prize contest because it wasn't amongst two selected submissions.\n\nSame for us. Additionally I don't see the value added for the organizers. If they want to implement a fast and accurate model in production, in my opinion it would have been better to allow for model compression as done in speech recognition challenge, and one could simply compress the top ensembles into one fast model. And even if model compression would be allowed, the rule of +/- 1 absolute percentage point seems odd. With same speed it would prefer a 0.51 -> 0.50 model over a 0.56 -> 0.54 model", "455880": "It was a memorable ensemble ;)", "456263": "We've considered the feedback about participation and eligible teams and decided to update and clarify the rules for participation and tie breaking in the special prize evaluation.\n\n- The top 1000 (1-1000) teams will now be eligible to participate in the special prize evaluation.\n- Beyond submitting the zip-file, a participating team must also fill in our [survey](https://docs.google.com/forms/d/e/1FAIpQLScFRTpX57XyW0nn0K1NVfk8oCQEMd9wD_9ALwlMD3yN4cDoVA/viewform?usp=sf_link%20%27survey%27) about the main competition. The survey deadline (January 16, 2019) as posted in the survey discussion [thread](https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/77367) is only applied for teams that want to participate as co-authors in the paper. For the special prize evaluation the survey can be filled in until January 25, 2019.\n- Where more than one team completes the evaluation with the same time, the team with the higher macro F1 score in the special prize evaluation will be preferred and win the tie. It doesn't matter how close a team's F1 score in the special prize evaluation is to the team's F1 score in the main competition, as long as it's within +/- 1 absolute percentage point.", "456268": "No, the same model must be used as submitted in the main competition.", "456269": "It's the private leaderboard.", "456431": "Thank you for considering the feedback. Again, i would like to bring to your attention the fact that this is actually against your best interests. Teams that submitted 14 models ensamble with 8 tta (:)) might have a really good single model that might fit the docker environment but it won't be within 1 percentage point. Why not just remove this condition, put some absolute barrier on the f1 (derived from your current solution) and let us do our bestest?", "456780": "Since it was possible to mark two submissins for evaluation, a team could submit two different approaches, eg ensemble and singel model, in the main competition.", "456810": "Since it was possible to mark two submissins for evaluation, a team could submit two different approaches, eg ensemble and singel model, in the main competition.\nYes, teams **could** do it, but nobody actually did it. It is useless to select single model in the main competition since ensemble is more stable and usually more accurate.", "456820": "> but nobody actually did it\n\napart from @bestfitting :D", "456823": "Yeah, probably :)", "458694": "martinhjelmare, will the docker test server have the integrated intel gpu? what kind?", "458696": "And yet again, I would like to comment how strange the requirement of being within +-1 of the f1 score of the competition is not realistic. Many teams (mine included) used a variable th based on class distribution. This mean that after we predict the whole test dataset, we select a th that will allow the relative number of entries from this class. Naturally, this can't be done prediction by prediction.", "458949": "also, the evaluation matrix is a tad unclear here. is speed part of it if its under 1h? or is it f1 like the competition? and what about leak data?", "459178": "just to make it easier for us, can you share the exact parameter you are going to run the docker image with? probably with 4096mb memory and 2 cores, but would be nice to know exactly how its going to be run.", "459360": "No. The test server won't have an integrated intel gpu. To allow more teams to participate we have to scale up to a server, and simplify hardware environment requirements.", "459364": "Speed (time) is the primary evaluation criteria. If there's a tie, higher f1 score wins.", "459372": "We can't share exact run parameters at this time. We're aiming to run the evaluation on a node with 2 cores and 4096 MB memory.", "459541": "Is using leak data allowed? It doesn't make any sense but it was used in the competition...", "460555": "And one more question.... 25th of Jan what time zone lol", "460957": "The deadline for submission will be Jan 25, 2019 at 11:59 p.m. PST.", "470665": "We are now finished with evaluation and are happy to announce the winning team:\n\nCongratulations to team Protein Shake! Their average prediction time per FOV was 67 ms.", "470929": "Whooohooo!!!! This is great!!!!\nThanks for doing this real life special prize. I enjoyed tackling the constraints very much." }, "source": "meta" }