How Are the Problem-Makers Going to Assess the Quality of Forecasts?\nThe problem of predicting laboratory earthquakes proposed by LAN staff for the Kaggle-competition is interesting and intriguing. However, it remains unclear the question of assessing the accuracy of forecasts and methods of comparison of forecasts proposed by the participants of the competition.\nIn fact, we are dealing with the problem of predicting a continuous random variable. The probability of an absolutely accurate prediction for at least one data segment is almost zero, regardless of the prediction algorithm. Nevertheless, the Directors expect point predictions as the responses. A number of questions arise.\n1) Is the quality of a single forecast estimated by the value of the absolute deviation from the unknown exact time before the earthquake?\n2) How is the quality of a single forecast calculated (for one data segment)?\n3) What is the formula to calculate the percentage of correct predictions?\n4) How are the forecasts compared?
\n\nThe answers to these questions are important both for the transparency of the competition and for the development of neural network learning algorithms. It would be desirable for the problem-makers to explicitly state the formula for the total (for all data segments) prediction error.
", "messages": [ { "id": "457266", "postDate": "01/17/2019 07:06:56", "content": "How Are the Problem-Makers Going to Assess the Quality of Forecasts?\nThe problem of predicting laboratory earthquakes proposed by LAN staff for the Kaggle-competition is interesting and intriguing. However, it remains unclear the question of assessing the accuracy of forecasts and methods of comparison of forecasts proposed by the participants of the competition.\nIn fact, we are dealing with the problem of predicting a continuous random variable. The probability of an absolutely accurate prediction for at least one data segment is almost zero, regardless of the prediction algorithm. Nevertheless, the Directors expect point predictions as the responses. A number of questions arise.\n1) Is the quality of a single forecast estimated by the value of the absolute deviation from the unknown exact time before the earthquake?\n2) How is the quality of a single forecast calculated (for one data segment)?\n3) What is the formula to calculate the percentage of correct predictions?\n4) How are the forecasts compared?
\n\nThe answers to these questions are important both for the transparency of the competition and for the development of neural network learning algorithms. It would be desirable for the problem-makers to explicitly state the formula for the total (for all data segments) prediction error.
", "rawMarkdown": "How Are the Problem-Makers Going to Assess the Quality of Forecasts?\nThe problem of predicting laboratory earthquakes proposed by LAN staff for the Kaggle-competition is interesting and intriguing. However, it remains unclear the question of assessing the accuracy of forecasts and methods of comparison of forecasts proposed by the participants of the competition.\nIn fact, we are dealing with the problem of predicting a continuous random variable. The probability of an absolutely accurate prediction for at least one data segment is almost zero, regardless of the prediction algorithm. Nevertheless, the Directors expect point predictions as the responses. A number of questions arise.\n1)\tIs the quality of a single forecast estimated by the value of the absolute deviation from the unknown exact time before the earthquake?\n2)\tHow is the quality of a single forecast calculated (for one data segment)?\n3)\t What is the formula to calculate the percentage of correct predictions?\n4)\tHow are the forecasts compared?\n\nThe answers to these questions are important both for the transparency of the competition and for the development of neural network learning algorithms. It would be desirable for the problem-makers to explicitly state the formula for the total (for all data segments) prediction error.", "votes": null }, { "id": "458360", "postDate": "01/19/2019 12:49:19", "content": "https://www.kaggle.com/c/LANL-Earthquake-Prediction#evaluation
", "rawMarkdown": "https://www.kaggle.com/c/LANL-Earthquake-Prediction#evaluation", "votes": null } ], "comments": [ { "id": 458360, "author_name": "danjel", "author_url": "", "post_date": "01/19/2019 12:49:19", "content": "https://www.kaggle.com/c/LANL-Earthquake-Prediction#evaluation
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "457266": "How Are the Problem-Makers Going to Assess the Quality of Forecasts?\nThe problem of predicting laboratory earthquakes proposed by LAN staff for the Kaggle-competition is interesting and intriguing. However, it remains unclear the question of assessing the accuracy of forecasts and methods of comparison of forecasts proposed by the participants of the competition.\nIn fact, we are dealing with the problem of predicting a continuous random variable. The probability of an absolutely accurate prediction for at least one data segment is almost zero, regardless of the prediction algorithm. Nevertheless, the Directors expect point predictions as the responses. A number of questions arise.\n1)\tIs the quality of a single forecast estimated by the value of the absolute deviation from the unknown exact time before the earthquake?\n2)\tHow is the quality of a single forecast calculated (for one data segment)?\n3)\t What is the formula to calculate the percentage of correct predictions?\n4)\tHow are the forecasts compared?\n\nThe answers to these questions are important both for the transparency of the competition and for the development of neural network learning algorithms. It would be desirable for the problem-makers to explicitly state the formula for the total (for all data segments) prediction error.", "458360": "https://www.kaggle.com/c/LANL-Earthquake-Prediction#evaluation" }, "source": "meta" }