As we found out a while back, http://soostrc.comet.ucar.edu/data/grib/rap/ seems to be more than one server. I have been seeing the one starting on September 29 pretty much all along; usually waiting a day or two or simply closing the browser and refreshing the DNS cache a few times brings back one with data starting June 20 (as of yesterday). Right now refreshing DNS doesn't seem to help, but maybe it just means the other server is temporarily down.
\nOr not. It would of course be great if Kaggle could provide a torrent of the relevant RAP data, preferably not corrupted. ;)
\nThere is more info about RAPv2 at
\nhttp://www.nws.noaa.gov/om/notification/tin13-38rap.htm
\nincluding a link to the parallel feed at
\nhttp://www.ftp.ncep.noaa.gov/data/nccf/com/rap/para
\nI was hoping the update would just be on the backend (better physics model) with maybe more fields added, but I just downloaded a v2 file and my RAP parser can't handle it. :(
\nEDIT: Turns out things are not so bad, you just need to pay attention to filenames. Make sure that you grap RAPv2 files for native levels (so with "brgb" in the filename) and for 13 km resolution ("awp13") and they seem to be compatible with the current RAP format. Phew!
", "rawMarkdown": "", "votes": null }, { "id": "35915", "postDate": "12/06/2013 20:28:47", "content": "Good to see that the new RAP files will still be readable. Too bad the old ones for the current leaderboard are not available. It looks like they're gone for good...
", "rawMarkdown": "", "votes": null }, { "id": "35918", "postDate": "12/06/2013 21:34:17", "content": "So how can those starting the competition now (or recently) get RAP data? The torrent is for Jun/Jul (and possibly corrupt), and the server only has files starting 9/29.
\nIMO If any piece of data is permitted for use, it should be provided to everyone, not secret to those who happened to download/archive it before it was taken offline.
", "rawMarkdown": "", "votes": null }, { "id": "35923", "postDate": "12/06/2013 22:48:34", "content": "My exact thought too.
\nI think that a new temporary leaderboard with new days could somehow alleviate the problem, even though not entirely.
", "rawMarkdown": "", "votes": null }, { "id": "35924", "postDate": "12/06/2013 22:51:25", "content": "Just found a different server with a longer history: ftp://nomads.ncdc.noaa.gov/RAP/13km/
", "rawMarkdown": "", "votes": null }, { "id": "35926", "postDate": "12/06/2013 23:07:05", "content": "I found that this morning, but unfortunately those grib files are different from the ones allowed in the competition: some layers are missing, other are different. The file size is smaller too, but that could be due to grib2 compression support...
", "rawMarkdown": "", "votes": null }, { "id": "35927", "postDate": "12/06/2013 23:18:50", "content": "I believe the file size difference may be due to the grid size - RUC is the original contig US grid, while new RAP is expanded by ~3x area (includes Alaska etc). Sim runs using original RUC grid size. The file size difference matches this proportion...
\nWhat is different about the layers?
", "rawMarkdown": "", "votes": null }, { "id": "35932", "postDate": "12/07/2013 00:24:24", "content": "Hmm, seems grid size the same in both distributions, guess the data is different. Couldn't get the file to match up with a weather.txt.gz.
", "rawMarkdown": "", "votes": null }, { "id": "35944", "postDate": "12/07/2013 13:04:20", "content": "Well, for example in RAP grib files we have 50 hybrid layers for u and v wind components, while in RUC ones the closest match is given by 38 isobaric layers, which are not exactly the same thing. Not to mention that, technically, we're not allowed to use those files... =(
", "rawMarkdown": "", "votes": null }, { "id": "35984", "postDate": "12/09/2013 07:06:27", "content": "Any official word on this topic?
", "rawMarkdown": "", "votes": null }, { "id": "35993", "postDate": "12/09/2013 18:07:19", "content": "Presumably the organizers have the SOOSTRC files, as they used them to generate the weather.txt.gz files. It would be as simple as providing them on the data page or via a torrent...
", "rawMarkdown": "", "votes": null }, { "id": "36005", "postDate": "12/09/2013 22:25:03", "content": "[quote=Brian K;35993]
\nPresumably the organizers have the SOOSTRC files, as they used them to generate the weather.txt.gz files. It would be as simple as providing them on the data page or via a torrent...
\n[/quote]
\nNot necessarily. The organizers would only need the last RAP analysis (0-hour forecast) before the cutoff time to generate the publicly disseminated weather file. They would also need several post-cutoff analyses to generate the weather file that is used behind the scenes to score the leaderboard submissions. However, we'd be cheating if we used any of those same post-cutoff analyses. It's the N-hour forecasts (for N > 0) that are relevant to us.
\nTo be clear: suppose the cutoff time for a given day is 18:17 UTC. The publicly released weather_{date}_1817.txt.gz file would be based on the 18z + 0 RAP forecast. The weather file provided to the flight simulator on Kaggle's evaluation server would also be derived from 19z + 0, 20z + 0, etc. But in our models we are only permitted to use 18z + 1, 18z + 2, etc.
\nHopefully the organizers bothered to download all of the RAP forecasts, not just the 0-hour ones. To ensure a fair competition, one of the following four courses of action needs to be taken:
\n(1) Make all Sep. 11-24 RAP forecasts available on the competition data page.
\n(2) Start over with a new leaderboard for a different range of dates, and make a permanent archive of the RAP data for that period.
\n(3) Allow competitors to use the slightly different RAP files available from the longer archive at http://nomads.ncdc.noaa.gov/data/rap130/ EDIT: After researching it further, this option is a bad idea. Not only is it impossible to exactly reconstruct data on the hybrid levels, but additionally some fields might be entirely missing from these other files. Competitors who possess the hybrid-coordinate files would still be at an unfair advantage.
\n(4) Disallow all use of RAP data.
\nI eagerly await clarification on this issue.
", "rawMarkdown": "", "votes": null }, { "id": "36006", "postDate": "12/09/2013 22:33:46", "content": "True, though even option 3 is disagreeable as unfair -- it is an unfair advantage to those who happened to acquire the once-available hybrid RAP files (advantage of less work to build a Z-layer interpolation model/allowing more time to work on other aspects of their model, possibly more accurate forecasts that align better with the future weather.txt.gz).
", "rawMarkdown": "", "votes": null }, { "id": "36007", "postDate": "12/09/2013 22:36:59", "content": "And, the longer the leaderboard/data possession exists in this state is an incremental advantage to those possessing the hybrid RAP files -- they are allowed more time/submissions to build/test their model using weather data and validate/receive score against the test set.
", "rawMarkdown": "", "votes": null }, { "id": "36038", "postDate": "12/10/2013 14:00:40", "content": "Would be really great if there could be an official answer, which data can be used, and if so, where to find it!
", "rawMarkdown": "", "votes": null }, { "id": "36040", "postDate": "12/10/2013 15:11:48", "content": "[quote=Doug Koch;36005]
\nNot necessarily. The organizers would only need the last RAP analysis (0-hour forecast) before the cutoff time to generate the publicly disseminated weather file. They would also need several post-cutoff analyses to generate the weather file that is used behind the scenes to score the leaderboard submissions. However, we'd be cheating if we used any of those same post-cutoff analyses. It's the N-hour forecasts (for N > 0) that are relevant to us.
\nTo be clear: suppose the cutoff time for a given day is 18:17 UTC. The publicly released weather_{date}_1817.txt.gz file would be based on the 18z + 0 RAP forecast. The weather file provided to the flight simulator on Kaggle's evaluation server would also be derived from 19z + 0, 20z + 0, etc. But in our models we are only permitted to use 18z + 1, 18z + 2, etc.
\nHopefully the organizers bothered to download all of the RAP forecasts, not just the 0-hour ones. To ensure a fair competition, one of the following four courses of action needs to be taken:
\n(1) Make all Sep. 11-24 RAP forecasts available on the competition data page.
\n(2) Start over with a new leaderboard for a different range of dates, and make a permanent archive of the RAP data for that period.
\n(3) Allow competitors to use the slightly different RAP files available from the longer archive at http://nomads.ncdc.noaa.gov/data/rap130/ EDIT: After researching it further, this option is a bad idea. Not only is it impossible to exactly reconstruct data on the hybrid levels, but additionally some fields might be entirely missing from these other files. Competitors who possess the hybrid-coordinate files would still be at an unfair advantage.
\n(4) Disallow all use of RAP data.
\nI eagerly await clarification on this issue.
\n[/quote]
\nPoint 4 seems a bit radical, but it would mean less problems in a general way, both now and in the future. Unfortunately this would mean a new leaderboard is necessary anyway, in case someone - legally - used RAP files for the current one.
", "rawMarkdown": "", "votes": null }, { "id": "36041", "postDate": "12/10/2013 18:06:20", "content": "Alternative #4 would mean backtracking on http://www.gequest.com/c/flight2-main/forums/t/6239/which-files-are-going-to-be-provided/35050#post35050 , posted 18 days ago. I have based all my decisions since then on the availability of RAP data, so a U-turn now would make me a little cranky.
", "rawMarkdown": "", "votes": null }, { "id": "36101", "postDate": "12/11/2013 22:31:28", "content": "Thanks to everyone for pointing this issue out.
\nWhile we have a copy of all the GRIB2 files relevant for running the Simulator on the current leaderboard data set, we do not have a copy of all the GRIB2 files that NOAA released over the corresponding period. We have reached out to NOAA and asked them to either put the remaining files back up or transfer them to us, but we have not yet heard back. We were not expecting these files to be taken down so soon.
\nWe are releasing the final hour of GRIB2 files published before the cutoff date on each day of the time period (the wind data for the first hour of the simulator was processed from these files). The previously released weather_*.txt.gz files contained the wind data processed from these GRIB2 files.
\nIf anyone downloaded a complete copy of all GRIB2 files relevant for the current leaderboard period (September 11-24, 2013), please let us know by replying here or contacting me directly. If so, they can be transferred to Kaggle via kaggle.com/files/drop and we'll release those as a torrent.
", "rawMarkdown": "", "votes": null }, { "id": "36104", "postDate": "12/11/2013 23:31:55", "content": "How were you going to verify that someone was using 'approved' rap forecasts without downloading them yourself?
", "rawMarkdown": "", "votes": null }, { "id": "36105", "postDate": "12/11/2013 23:34:36", "content": "And now, what's to stop some competitor from creating some throw-away account, taking rap forecasts, adding some level of noise, and saying "here"? (While using the actual forecasts for their submission)
\nGoes back to my previous post I guess -- without having the files yourself/knowing which forecasts are approved, how were/are you going to verify?
", "rawMarkdown": "", "votes": null }, { "id": "36106", "postDate": "12/12/2013 00:31:46", "content": "[quote=Brian K;36105]
\nAnd now, what's to stop some competitor from creating some throw-away account, taking rap forecasts, adding some level of noise, and saying "here"? (While using the actual forecasts for their submission)
\nGoes back to my previous post I guess -- without having the files yourself/knowing which forecasts are approved, how were/are you going to verify?
\n[/quote]This isn't relevant for the current leaderboard data, since no verification is done on that leaderboard and no prize money is awarded based on that leaderboard (note all future prize money is awarded based on the final evaluation leaderboard, the data for which doesn't exist yet).
", "rawMarkdown": "", "votes": null }, { "id": "36107", "postDate": "12/12/2013 00:35:48", "content": "Sure, for which the data isn't released yet, but one would hope the same policies/procedures are in place for the data acquisition, handling, and generation.
", "rawMarkdown": "", "votes": null }, { "id": "36108", "postDate": "12/12/2013 00:38:15", "content": "And I would say it still is relevant for current leaderboard -- someone providing false forecast data for this set could mislead competitors in the development process (ie about the value of weather information, leading people to believe "oh these forecasts are crap, i'm not going to bother with them in my model"). Even if it isn't used for any prizes awarded...
", "rawMarkdown": "", "votes": null }, { "id": "36140", "postDate": "12/13/2013 04:33:22", "content": "I have uploaded a zip file with cutoff hour and subsequent forecast RAPs for September
11 through 24. Sorry it took me so long to hack up an encoder and fake the data with
noise. :P I also named the file "sep11-18rap.zip" to really confuse you all.
Good news: RAP forecasts are now once again available all the way back to June 20! http://soostrc.comet.ucar.edu/data/grib/rap/
\nI'd urge the Kaggle admins to immediately download all September 11-24 forecasts in case they disappear again.
", "rawMarkdown": "", "votes": null }, { "id": "36235", "postDate": "12/17/2013 07:21:59", "content": "What the heck? They're offline once again...
\n(actually they're online, but the links are hidden and I get a 403 forbidden error page if I try to access them through a reconstructed link)
", "rawMarkdown": "", "votes": null }, { "id": "36237", "postDate": "12/17/2013 07:44:17", "content": "I got the files 10 minutes ago without any problem. So maybe some local problem with your computer/internet connection?
", "rawMarkdown": "", "votes": null }, { "id": "36238", "postDate": "12/17/2013 08:20:58", "content": "I tried both from behind a proxy and from a smartphone, but I'm back to the situation of a few days ago: the first day on the list i 2013-09-29.
", "rawMarkdown": "", "votes": null }, { "id": "36242", "postDate": "12/17/2013 09:19:52", "content": "Seems they are really gone ... Now I cannot see them anymore in the browser.
", "rawMarkdown": "", "votes": null }, { "id": "36243", "postDate": "12/17/2013 09:22:46", "content": "@Gabriele: I have at least all the GRIB files for September 11-24 forecasts at cut-off time. If you want them (7.4 GB) write me a short mail (I cannot contact other users directly because I have any kaggle points :)).
", "rawMarkdown": "", "votes": null }, { "id": "36247", "postDate": "12/17/2013 11:02:26", "content": "I wonder why Kaggle still hasn't posted a torrent of the files I uploaded last week.
\nAnyway, to reiterate: this on/off appearance of what seems to be at least two servers with different sets of RAP files is what I have been seeing for months. This time it just took a little longer than usual for the larger set to reappear.
", "rawMarkdown": "", "votes": null }, { "id": "36313", "postDate": "12/18/2013 15:05:20", "content": "You're probably right: today they seem to be available again.
", "rawMarkdown": "", "votes": null }, { "id": "36682", "postDate": "12/26/2013 23:23:31", "content": "Thanks to TecS for supplying a copy of the RAP files. Does anyone have the forecast RAP files for Sep 10th? I noticed that Kaggle has "Weather GRIB2 Files (Sept. 10th).zip" available for download, but this archive does not contain forecasts.
\nTo be clear, I am looking for the following files:
\n13091018.rap.t18z.awp130bgrbf01.grib2
\n13091018.rap.t18z.awp130bgrbf02.grib2
\n13091018.rap.t18z.awp130bgrbf03.grib2
\n13091018.rap.t18z.awp130bgrbf04.grib2
\n13091018.rap.t18z.awp130bgrbf05.grib2
\n13091018.rap.t18z.awp130bgrbf06.grib2
\n13091018.rap.t18z.awp130bgrbf07.grib2
\nI understand that these were previously available for download at http://soostrc.comet.ucar.edu/data/grib/rap/.
\nBy the way, the file "Weather Forecast GRIB2 Files (Sept 11-24) - Provided by TecS.zip" that Kaggle has made available for download, appears to be corrupt. I had to use the fix option in the zip utility to salvage the data. Haven't checked if the data is actually usable. I tried to download from two different machines and got the same result. The downloaded file is 8227061008 bytes long and has an md5 sum of e92d80340e68452675badec15564eaa6.
\nAdmins, please check... It is no fun to download corrupt 8GB files.
\n\n
\n
\n
", "rawMarkdown": "", "votes": null }, { "id": "36714", "postDate": "12/27/2013 17:18:29", "content": "
I still have the zip archive I uploaded, just checked, and size & md5sum are as you reported. So nothing seems to have been corrupted in transfer.
\nWindows Explorer (8.1, 64 bit) has no problem opening and navigating it. It was created using "Free Zip Opener" ( http://www.radzipper.com/ ). Maybe if you try using that it will work for you? Only thing I can think of is that older and/or 32 bit zip utilities might choke on archives larger than 2 GB.
\nI have now also uploaded a zip file with 0th hour weather and 7 subsequent hourly forecasts for September 10 to
\nhttp://bayfiles.net/file/12AyX/Ujak6Z/sep10rap.zip
\nmd5sum for this one is
\nbed765eb462424e99cb6ce7abf6203e4
\nsize is
\n248,876,189 bytes
\n(so not large enough to be a problem even with old unzippers).
", "rawMarkdown": "", "votes": null }, { "id": "36725", "postDate": "12/27/2013 19:09:24", "content": "Much appreciated! I am pulling down the file as I type.
\nI had trouble with the other zip file on both Mac OS and Ubuntu. The error message was:
\n"start of central directory not found; zipfile corrupt."
\nDidn't try Windows.
\n\n
UPDATE: It was the unzip utility on Mac OS and Ubuntu that had problems with the zip file. I just tried 7za instead and it worked fine.
", "rawMarkdown": "", "votes": null }, { "id": "36730", "postDate": "12/27/2013 20:26:45", "content": "Seems consistent with http://superuser.com/questions/114011/extract-large-zip-file-50-gb-on-mac-os-x/249689#249689
\n", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 35907, "author_name": "hyperborean", "author_url": "", "post_date": "12/06/2013 12:14:17", "content": "
As we found out a while back, http://soostrc.comet.ucar.edu/data/grib/rap/ seems to be more than one server. I have been seeing the one starting on September 29 pretty much all along; usually waiting a day or two or simply closing the browser and refreshing the DNS cache a few times brings back one with data starting June 20 (as of yesterday). Right now refreshing DNS doesn't seem to help, but maybe it just means the other server is temporarily down.
\nOr not. It would of course be great if Kaggle could provide a torrent of the relevant RAP data, preferably not corrupted. ;)
\nThere is more info about RAPv2 at
\nhttp://www.nws.noaa.gov/om/notification/tin13-38rap.htm
\nincluding a link to the parallel feed at
\nhttp://www.ftp.ncep.noaa.gov/data/nccf/com/rap/para
\nI was hoping the update would just be on the backend (better physics model) with maybe more fields added, but I just downloaded a v2 file and my RAP parser can't handle it. :(
\nEDIT: Turns out things are not so bad, you just need to pay attention to filenames. Make sure that you grap RAPv2 files for native levels (so with "brgb" in the filename) and for 13 km resolution ("awp13") and they seem to be compatible with the current RAP format. Phew!
", "votes": null, "replies": [] }, { "id": 35915, "author_name": "gabrieleseppi", "author_url": "", "post_date": "12/06/2013 20:28:47", "content": "Good to see that the new RAP files will still be readable. Too bad the old ones for the current leaderboard are not available. It looks like they're gone for good...
", "votes": null, "replies": [] }, { "id": 35918, "author_name": "briank2", "author_url": "", "post_date": "12/06/2013 21:34:17", "content": "So how can those starting the competition now (or recently) get RAP data? The torrent is for Jun/Jul (and possibly corrupt), and the server only has files starting 9/29.
\nIMO If any piece of data is permitted for use, it should be provided to everyone, not secret to those who happened to download/archive it before it was taken offline.
", "votes": null, "replies": [] }, { "id": 35923, "author_name": "gabrieleseppi", "author_url": "", "post_date": "12/06/2013 22:48:34", "content": "My exact thought too.
\nI think that a new temporary leaderboard with new days could somehow alleviate the problem, even though not entirely.
", "votes": null, "replies": [] }, { "id": 35924, "author_name": "briank2", "author_url": "", "post_date": "12/06/2013 22:51:25", "content": "Just found a different server with a longer history: ftp://nomads.ncdc.noaa.gov/RAP/13km/
", "votes": null, "replies": [] }, { "id": 35926, "author_name": "gabrieleseppi", "author_url": "", "post_date": "12/06/2013 23:07:05", "content": "I found that this morning, but unfortunately those grib files are different from the ones allowed in the competition: some layers are missing, other are different. The file size is smaller too, but that could be due to grib2 compression support...
", "votes": null, "replies": [] }, { "id": 35927, "author_name": "briank2", "author_url": "", "post_date": "12/06/2013 23:18:50", "content": "I believe the file size difference may be due to the grid size - RUC is the original contig US grid, while new RAP is expanded by ~3x area (includes Alaska etc). Sim runs using original RUC grid size. The file size difference matches this proportion...
\nWhat is different about the layers?
", "votes": null, "replies": [] }, { "id": 35932, "author_name": "briank2", "author_url": "", "post_date": "12/07/2013 00:24:24", "content": "Hmm, seems grid size the same in both distributions, guess the data is different. Couldn't get the file to match up with a weather.txt.gz.
", "votes": null, "replies": [] }, { "id": 35944, "author_name": "gabrieleseppi", "author_url": "", "post_date": "12/07/2013 13:04:20", "content": "Well, for example in RAP grib files we have 50 hybrid layers for u and v wind components, while in RUC ones the closest match is given by 38 isobaric layers, which are not exactly the same thing. Not to mention that, technically, we're not allowed to use those files... =(
", "votes": null, "replies": [] }, { "id": 35984, "author_name": "gabrieleseppi", "author_url": "", "post_date": "12/09/2013 07:06:27", "content": "Any official word on this topic?
", "votes": null, "replies": [] }, { "id": 35993, "author_name": "briank2", "author_url": "", "post_date": "12/09/2013 18:07:19", "content": "Presumably the organizers have the SOOSTRC files, as they used them to generate the weather.txt.gz files. It would be as simple as providing them on the data page or via a torrent...
", "votes": null, "replies": [] }, { "id": 36005, "author_name": "dougkoch", "author_url": "", "post_date": "12/09/2013 22:25:03", "content": "[quote=Brian K;35993]
\nPresumably the organizers have the SOOSTRC files, as they used them to generate the weather.txt.gz files. It would be as simple as providing them on the data page or via a torrent...
\n[/quote]
\nNot necessarily. The organizers would only need the last RAP analysis (0-hour forecast) before the cutoff time to generate the publicly disseminated weather file. They would also need several post-cutoff analyses to generate the weather file that is used behind the scenes to score the leaderboard submissions. However, we'd be cheating if we used any of those same post-cutoff analyses. It's the N-hour forecasts (for N > 0) that are relevant to us.
\nTo be clear: suppose the cutoff time for a given day is 18:17 UTC. The publicly released weather_{date}_1817.txt.gz file would be based on the 18z + 0 RAP forecast. The weather file provided to the flight simulator on Kaggle's evaluation server would also be derived from 19z + 0, 20z + 0, etc. But in our models we are only permitted to use 18z + 1, 18z + 2, etc.
\nHopefully the organizers bothered to download all of the RAP forecasts, not just the 0-hour ones. To ensure a fair competition, one of the following four courses of action needs to be taken:
\n(1) Make all Sep. 11-24 RAP forecasts available on the competition data page.
\n(2) Start over with a new leaderboard for a different range of dates, and make a permanent archive of the RAP data for that period.
\n(3) Allow competitors to use the slightly different RAP files available from the longer archive at http://nomads.ncdc.noaa.gov/data/rap130/ EDIT: After researching it further, this option is a bad idea. Not only is it impossible to exactly reconstruct data on the hybrid levels, but additionally some fields might be entirely missing from these other files. Competitors who possess the hybrid-coordinate files would still be at an unfair advantage.
\n(4) Disallow all use of RAP data.
\nI eagerly await clarification on this issue.
", "votes": null, "replies": [] }, { "id": 36006, "author_name": "briank2", "author_url": "", "post_date": "12/09/2013 22:33:46", "content": "True, though even option 3 is disagreeable as unfair -- it is an unfair advantage to those who happened to acquire the once-available hybrid RAP files (advantage of less work to build a Z-layer interpolation model/allowing more time to work on other aspects of their model, possibly more accurate forecasts that align better with the future weather.txt.gz).
", "votes": null, "replies": [] }, { "id": 36007, "author_name": "briank2", "author_url": "", "post_date": "12/09/2013 22:36:59", "content": "And, the longer the leaderboard/data possession exists in this state is an incremental advantage to those possessing the hybrid RAP files -- they are allowed more time/submissions to build/test their model using weather data and validate/receive score against the test set.
", "votes": null, "replies": [] }, { "id": 36038, "author_name": "derdasmachenmuss", "author_url": "", "post_date": "12/10/2013 14:00:40", "content": "Would be really great if there could be an official answer, which data can be used, and if so, where to find it!
", "votes": null, "replies": [] }, { "id": 36040, "author_name": "gabrieleseppi", "author_url": "", "post_date": "12/10/2013 15:11:48", "content": "[quote=Doug Koch;36005]
\nNot necessarily. The organizers would only need the last RAP analysis (0-hour forecast) before the cutoff time to generate the publicly disseminated weather file. They would also need several post-cutoff analyses to generate the weather file that is used behind the scenes to score the leaderboard submissions. However, we'd be cheating if we used any of those same post-cutoff analyses. It's the N-hour forecasts (for N > 0) that are relevant to us.
\nTo be clear: suppose the cutoff time for a given day is 18:17 UTC. The publicly released weather_{date}_1817.txt.gz file would be based on the 18z + 0 RAP forecast. The weather file provided to the flight simulator on Kaggle's evaluation server would also be derived from 19z + 0, 20z + 0, etc. But in our models we are only permitted to use 18z + 1, 18z + 2, etc.
\nHopefully the organizers bothered to download all of the RAP forecasts, not just the 0-hour ones. To ensure a fair competition, one of the following four courses of action needs to be taken:
\n(1) Make all Sep. 11-24 RAP forecasts available on the competition data page.
\n(2) Start over with a new leaderboard for a different range of dates, and make a permanent archive of the RAP data for that period.
\n(3) Allow competitors to use the slightly different RAP files available from the longer archive at http://nomads.ncdc.noaa.gov/data/rap130/ EDIT: After researching it further, this option is a bad idea. Not only is it impossible to exactly reconstruct data on the hybrid levels, but additionally some fields might be entirely missing from these other files. Competitors who possess the hybrid-coordinate files would still be at an unfair advantage.
\n(4) Disallow all use of RAP data.
\nI eagerly await clarification on this issue.
\n[/quote]
\nPoint 4 seems a bit radical, but it would mean less problems in a general way, both now and in the future. Unfortunately this would mean a new leaderboard is necessary anyway, in case someone - legally - used RAP files for the current one.
", "votes": null, "replies": [] }, { "id": 36041, "author_name": "hyperborean", "author_url": "", "post_date": "12/10/2013 18:06:20", "content": "Alternative #4 would mean backtracking on http://www.gequest.com/c/flight2-main/forums/t/6239/which-files-are-going-to-be-provided/35050#post35050 , posted 18 days ago. I have based all my decisions since then on the availability of RAP data, so a U-turn now would make me a little cranky.
", "votes": null, "replies": [] }, { "id": 36101, "author_name": "joycenv", "author_url": "", "post_date": "12/11/2013 22:31:28", "content": "Thanks to everyone for pointing this issue out.
\nWhile we have a copy of all the GRIB2 files relevant for running the Simulator on the current leaderboard data set, we do not have a copy of all the GRIB2 files that NOAA released over the corresponding period. We have reached out to NOAA and asked them to either put the remaining files back up or transfer them to us, but we have not yet heard back. We were not expecting these files to be taken down so soon.
\nWe are releasing the final hour of GRIB2 files published before the cutoff date on each day of the time period (the wind data for the first hour of the simulator was processed from these files). The previously released weather_*.txt.gz files contained the wind data processed from these GRIB2 files.
\nIf anyone downloaded a complete copy of all GRIB2 files relevant for the current leaderboard period (September 11-24, 2013), please let us know by replying here or contacting me directly. If so, they can be transferred to Kaggle via kaggle.com/files/drop and we'll release those as a torrent.
", "votes": null, "replies": [] }, { "id": 36104, "author_name": "briank2", "author_url": "", "post_date": "12/11/2013 23:31:55", "content": "How were you going to verify that someone was using 'approved' rap forecasts without downloading them yourself?
", "votes": null, "replies": [] }, { "id": 36105, "author_name": "briank2", "author_url": "", "post_date": "12/11/2013 23:34:36", "content": "And now, what's to stop some competitor from creating some throw-away account, taking rap forecasts, adding some level of noise, and saying "here"? (While using the actual forecasts for their submission)
\nGoes back to my previous post I guess -- without having the files yourself/knowing which forecasts are approved, how were/are you going to verify?
", "votes": null, "replies": [] }, { "id": 36106, "author_name": "benhamner", "author_url": "", "post_date": "12/12/2013 00:31:46", "content": "[quote=Brian K;36105]
\nAnd now, what's to stop some competitor from creating some throw-away account, taking rap forecasts, adding some level of noise, and saying "here"? (While using the actual forecasts for their submission)
\nGoes back to my previous post I guess -- without having the files yourself/knowing which forecasts are approved, how were/are you going to verify?
\n[/quote]This isn't relevant for the current leaderboard data, since no verification is done on that leaderboard and no prize money is awarded based on that leaderboard (note all future prize money is awarded based on the final evaluation leaderboard, the data for which doesn't exist yet).
", "votes": null, "replies": [] }, { "id": 36107, "author_name": "briank2", "author_url": "", "post_date": "12/12/2013 00:35:48", "content": "Sure, for which the data isn't released yet, but one would hope the same policies/procedures are in place for the data acquisition, handling, and generation.
", "votes": null, "replies": [] }, { "id": 36108, "author_name": "briank2", "author_url": "", "post_date": "12/12/2013 00:38:15", "content": "And I would say it still is relevant for current leaderboard -- someone providing false forecast data for this set could mislead competitors in the development process (ie about the value of weather information, leading people to believe "oh these forecasts are crap, i'm not going to bother with them in my model"). Even if it isn't used for any prizes awarded...
", "votes": null, "replies": [] }, { "id": 36140, "author_name": "hyperborean", "author_url": "", "post_date": "12/13/2013 04:33:22", "content": "I have uploaded a zip file with cutoff hour and subsequent forecast RAPs for September
11 through 24. Sorry it took me so long to hack up an encoder and fake the data with
noise. :P I also named the file "sep11-18rap.zip" to really confuse you all.
Good news: RAP forecasts are now once again available all the way back to June 20! http://soostrc.comet.ucar.edu/data/grib/rap/
\nI'd urge the Kaggle admins to immediately download all September 11-24 forecasts in case they disappear again.
", "votes": null, "replies": [] }, { "id": 36235, "author_name": "gabrieleseppi", "author_url": "", "post_date": "12/17/2013 07:21:59", "content": "What the heck? They're offline once again...
\n(actually they're online, but the links are hidden and I get a 403 forbidden error page if I try to access them through a reconstructed link)
", "votes": null, "replies": [] }, { "id": 36237, "author_name": "derdasmachenmuss", "author_url": "", "post_date": "12/17/2013 07:44:17", "content": "I got the files 10 minutes ago without any problem. So maybe some local problem with your computer/internet connection?
", "votes": null, "replies": [] }, { "id": 36238, "author_name": "gabrieleseppi", "author_url": "", "post_date": "12/17/2013 08:20:58", "content": "I tried both from behind a proxy and from a smartphone, but I'm back to the situation of a few days ago: the first day on the list i 2013-09-29.
", "votes": null, "replies": [] }, { "id": 36242, "author_name": "derdasmachenmuss", "author_url": "", "post_date": "12/17/2013 09:19:52", "content": "Seems they are really gone ... Now I cannot see them anymore in the browser.
", "votes": null, "replies": [] }, { "id": 36243, "author_name": "derdasmachenmuss", "author_url": "", "post_date": "12/17/2013 09:22:46", "content": "@Gabriele: I have at least all the GRIB files for September 11-24 forecasts at cut-off time. If you want them (7.4 GB) write me a short mail (I cannot contact other users directly because I have any kaggle points :)).
", "votes": null, "replies": [] }, { "id": 36247, "author_name": "hyperborean", "author_url": "", "post_date": "12/17/2013 11:02:26", "content": "I wonder why Kaggle still hasn't posted a torrent of the files I uploaded last week.
\nAnyway, to reiterate: this on/off appearance of what seems to be at least two servers with different sets of RAP files is what I have been seeing for months. This time it just took a little longer than usual for the larger set to reappear.
", "votes": null, "replies": [] }, { "id": 36313, "author_name": "gabrieleseppi", "author_url": "", "post_date": "12/18/2013 15:05:20", "content": "You're probably right: today they seem to be available again.
", "votes": null, "replies": [] }, { "id": 36682, "author_name": "anlthms", "author_url": "", "post_date": "12/26/2013 23:23:31", "content": "Thanks to TecS for supplying a copy of the RAP files. Does anyone have the forecast RAP files for Sep 10th? I noticed that Kaggle has "Weather GRIB2 Files (Sept. 10th).zip" available for download, but this archive does not contain forecasts.
\nTo be clear, I am looking for the following files:
\n13091018.rap.t18z.awp130bgrbf01.grib2
\n13091018.rap.t18z.awp130bgrbf02.grib2
\n13091018.rap.t18z.awp130bgrbf03.grib2
\n13091018.rap.t18z.awp130bgrbf04.grib2
\n13091018.rap.t18z.awp130bgrbf05.grib2
\n13091018.rap.t18z.awp130bgrbf06.grib2
\n13091018.rap.t18z.awp130bgrbf07.grib2
\nI understand that these were previously available for download at http://soostrc.comet.ucar.edu/data/grib/rap/.
\nBy the way, the file "Weather Forecast GRIB2 Files (Sept 11-24) - Provided by TecS.zip" that Kaggle has made available for download, appears to be corrupt. I had to use the fix option in the zip utility to salvage the data. Haven't checked if the data is actually usable. I tried to download from two different machines and got the same result. The downloaded file is 8227061008 bytes long and has an md5 sum of e92d80340e68452675badec15564eaa6.
\nAdmins, please check... It is no fun to download corrupt 8GB files.
\n\n
\n
\n
", "votes": null, "replies": [] }, { "id": 36714, "author_name": "hyperborean", "author_url": "", "post_date": "12/27/2013 17:18:29", "content": "
I still have the zip archive I uploaded, just checked, and size & md5sum are as you reported. So nothing seems to have been corrupted in transfer.
\nWindows Explorer (8.1, 64 bit) has no problem opening and navigating it. It was created using "Free Zip Opener" ( http://www.radzipper.com/ ). Maybe if you try using that it will work for you? Only thing I can think of is that older and/or 32 bit zip utilities might choke on archives larger than 2 GB.
\nI have now also uploaded a zip file with 0th hour weather and 7 subsequent hourly forecasts for September 10 to
\nhttp://bayfiles.net/file/12AyX/Ujak6Z/sep10rap.zip
\nmd5sum for this one is
\nbed765eb462424e99cb6ce7abf6203e4
\nsize is
\n248,876,189 bytes
\n(so not large enough to be a problem even with old unzippers).
", "votes": null, "replies": [] }, { "id": 36725, "author_name": "anlthms", "author_url": "", "post_date": "12/27/2013 19:09:24", "content": "Much appreciated! I am pulling down the file as I type.
\nI had trouble with the other zip file on both Mac OS and Ubuntu. The error message was:
\n"start of central directory not found; zipfile corrupt."
\nDidn't try Windows.
\n\n
UPDATE: It was the unzip utility on Mac OS and Ubuntu that had problems with the zip file. I just tried 7za instead and it worked fine.
", "votes": null, "replies": [] }, { "id": 36730, "author_name": "hyperborean", "author_url": "", "post_date": "12/27/2013 20:26:45", "content": "Seems consistent with http://superuser.com/questions/114011/extract-large-zip-file-50-gb-on-mac-os-x/249689#249689
\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "35904": "", "35907": "", "35915": "", "35918": "", "35923": "", "35924": "", "35926": "", "35927": "", "35932": "", "35944": "", "35984": "", "35993": "", "36005": "", "36006": "", "36007": "", "36038": "", "36040": "", "36041": "", "36101": "", "36104": "", "36105": "", "36106": "", "36107": "", "36108": "", "36140": "", "36193": "", "36235": "", "36237": "", "36238": "", "36242": "", "36243": "", "36247": "", "36313": "", "36682": "", "36714": "", "36725": "", "36730": "" }, "source": "meta" }