{"cells":[{"metadata":{},"cell_type":"markdown","source":"# There are between 4 and 5 songs per recording on average.\n\nThe sample submission get a score of 0.185.  What can we get out of this?\n\nFrist, let's import what we need. The competition metric is said to be similar to sklean `label_ranking_average_precision_score` except for rows weights.  IF we have a single row then they should be equal."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nfrom sklearn.metrics import label_ranking_average_precision_score\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"let's compute what a constant 0.5 submission, as in sample submission, would yield with rows of varying number of songs."},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = 0.5 * np.ones((1, 24))\nfor i in range(25):\n    y_true = np.zeros((1, 24))\n    y_true[0, :i] = 1\n    score = label_ranking_average_precision_score(y_true, y_pred)\n    print(i, '%0.3f' % score, y_true)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that sample submisison score of 0.185 lies between the scores with 4 or 5 targets equal to one.\n\nWait a minute.  Is the ordering of targets relevant?  No, according to the metric definition.  \n\nLet's check by reversing the order of targets."},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = 0.5 * np.ones((1, 24))\nfor i in range(25):\n    y_true = np.zeros((1, 24))\n    y_true[0, i:] = 1\n    score = label_ranking_average_precision_score(y_true, y_pred)\n    print(i, '%0.3f' % score, y_true)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that 0.185 still lies between the scores of row with 4 and 5 targets equal to one.\n\nHow can we use this information?  I don't know for now.  It would certainly be helpful if the metric was about rounded predictions, like in the Cornell birsdsong competition.  Here the ordering of predictions is what maters, and knowing how many should be close to 1 is not very helpful.  Yet, why not sharing this?\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}