{ "id": 3559, "title": "Ben: Code to create test set from training", "url": "/competitions/flight/discussion/3559", "author_name": "", "post_date": "2013-01-09T06:54:33.217Z", "votes": null, "comment_count": 2, "views": 2694, "content": "

Hi Ben,

\r\n

You were to post a mechanism to get test set from training. THere is no need for code from you - just tell us the logic that you use.

\r\n

For example, here is my R Code to create test from training:

\r\n

Here is what I am doing -  however I see that the CV score for the most recent value of ERA/EGA also differs from public leaderboard score. This means that there is some additional logic to code below

\r\n

 

\r\n

My Method:

\r\n

1. remove flights that do not arrive/depart in US

\r\n

2. remove flights that have an invalid form of datetime - either arrival or departure date

\r\n

3. only flights that are in air at that time

\r\n

4. remove flights whose gate_departure time is after runway_departure_time OR whose gate arrival is before runway arrival

\r\n

5. use the same cutoff days as in the file

\r\n

 

\r\n

 

\r\n

What else is needed to be done?

\r\n

Thanks
\r\nkiran 

\r\n

 

\r\n
cutoff_days <- read.csv (paste (cv_path, 'days.csv', sep='/'), header=TRUE, stringsAsFactors = FALSE)
\tcutoff_days$cutoff_time <-  as.POSIXct (cutoff_days$selected_cutoff_time, tz="UTC", format = "%Y-%m-%d %H:%M:%S", origin="1970-1-1"  )

\t#myNms <- c ("flight_history_id", "departure_airport_code", "arrival_airport_code", \t"published_departure",\t"published_arrival",\t"scheduled_gate_departure",\t"scheduled_gate_arrival",\t"scheduled_runway_departure",   "scheduled_runway_arrival", "actual_gate_arrival", "actual_runway_arrival", "actual_gate_arrival_mins", "actual_runway_arrival_mins", "airline_icao_code","departure_airport_icao_code", "arrival_airport_icao_code", "creator_code", "scheduled_aircraft_type", "actual_aircraft_type", "scheduled_air_time", "actual_gate_departure", "actual_runway_departure")

\tmyNms <- c ("flight_history_id",          "departure_airport_code"   ,  "arrival_airport_code",       "published_departure"       , "published_arrival",          "scheduled_gate_departure"  , "scheduled_gate_arrival",     "scheduled_runway_departure",
 "scheduled_runway_arrival")

\tfor (myDate in test_dates) {
\t\tprint (paste ("for myDate", myDate, sep=":"))

\t\tbase_day_path <- paste (cv_path,myDate, sep="/")
\t\tmyList <- summarize_flight_history_day (base_day_path, test_day_path = NA)
\t\tflighthistory <- myList [[1]]

\t\t# remove flights that do not arrive and depart both in the US
\t\tflighthistory_departs_from_us <- flighthistory$departure_airport_icao_code %in% us_icao_codes$icao_code
\t\tflighthistory_arrives_in_us <- flighthistory$arrival_airport_icao_code %in% us_icao_codes$icao_code
\t\tmyRows <- which ((which(flighthistory_arrives_in_us == TRUE) & which(flighthistory_departs_from_us == TRUE)) == TRUE)
\t\tflighthistory <- flighthistory[myRows,]

\t\t# remove flights that have an invalid form of datetime - either arrival or departure date
\t\tignoreRows <- c(which (is.na (flighthistory$actual_gate_departure) == TRUE)
\t\t, which (is.na (flighthistory$actual_gate_arrival) == TRUE)
\t\t, which (is.na (flighthistory$actual_runway_arrival) == TRUE)
\t\t, which (is.na (flighthistory$actual_runway_departure) == TRUE))
\t\tignoreRows <- unique (ignoreRows)

\t\tflighthistory <- flighthistory [-ignoreRows,]

\t\t# only flights that are in air at that time
\t\tflighthistory <- flighthistory [!is.na(flighthistory$actual_gate_arrival),]
\t\tflighthistory <- flighthistory [!is.na(flighthistory$actual_runway_arrival),]


\t\t# remove flights whose gate_departure time is after runway_departure_time OR whose gate arrival is before runway arrival
\t\tignoreRows <- c(which (flighthistory$actual_gate_departure > flighthistory$actual_runway_departure), which (flighthistory$actual_runway_arrival > flighthistory$actual_gate_arrival))
\t\tignoreRows <- unique (ignoreRows)

\t\tflighthistory <- flighthistory [-ignoreRows,]

\t\t

\t\tflighthistory$cutoff_time <- as.POSIXct (cutoff_days [cutoff_days$folder_name == myDate, "selected_cutoff_time"],  tz="UTC", format = "%Y-%m-%d %H:%M:%S", origin="1970-1-1"  )
\t\t
\t\ttestflighthistory <- flighthistory [flighthistory$actual_gate_arrival >= flighthistory$cutoff_time,]

 \t\t
\t\t
\t\twrite.csv (x=testflighthistory[,myNms]  ,file=paste (cv_path, myDate, "test_flights_new.csv", sep= "/"), row.names = FALSE)


\t}
", "messages": [ { "id": "19121", "postDate": "01/09/2013 06:54:33", "content": "

Hi Ben,

\r\n

You were to post a mechanism to get test set from training. THere is no need for code from you - just tell us the logic that you use.

\r\n

For example, here is my R Code to create test from training:

\r\n

Here is what I am doing -  however I see that the CV score for the most recent value of ERA/EGA also differs from public leaderboard score. This means that there is some additional logic to code below

\r\n

 

\r\n

My Method:

\r\n

1. remove flights that do not arrive/depart in US

\r\n

2. remove flights that have an invalid form of datetime - either arrival or departure date

\r\n

3. only flights that are in air at that time

\r\n

4. remove flights whose gate_departure time is after runway_departure_time OR whose gate arrival is before runway arrival

\r\n

5. use the same cutoff days as in the file

\r\n

 

\r\n

 

\r\n

What else is needed to be done?

\r\n

Thanks
\r\nkiran 

\r\n

 

\r\n
cutoff_days <- read.csv (paste (cv_path, 'days.csv', sep='/'), header=TRUE, stringsAsFactors = FALSE)
\tcutoff_days$cutoff_time <-  as.POSIXct (cutoff_days$selected_cutoff_time, tz="UTC", format = "%Y-%m-%d %H:%M:%S", origin="1970-1-1"  )

\t#myNms <- c ("flight_history_id", "departure_airport_code", "arrival_airport_code", \t"published_departure",\t"published_arrival",\t"scheduled_gate_departure",\t"scheduled_gate_arrival",\t"scheduled_runway_departure",   "scheduled_runway_arrival", "actual_gate_arrival", "actual_runway_arrival", "actual_gate_arrival_mins", "actual_runway_arrival_mins", "airline_icao_code","departure_airport_icao_code", "arrival_airport_icao_code", "creator_code", "scheduled_aircraft_type", "actual_aircraft_type", "scheduled_air_time", "actual_gate_departure", "actual_runway_departure")

\tmyNms <- c ("flight_history_id",          "departure_airport_code"   ,  "arrival_airport_code",       "published_departure"       , "published_arrival",          "scheduled_gate_departure"  , "scheduled_gate_arrival",     "scheduled_runway_departure",
 "scheduled_runway_arrival")

\tfor (myDate in test_dates) {
\t\tprint (paste ("for myDate", myDate, sep=":"))

\t\tbase_day_path <- paste (cv_path,myDate, sep="/")
\t\tmyList <- summarize_flight_history_day (base_day_path, test_day_path = NA)
\t\tflighthistory <- myList [[1]]

\t\t# remove flights that do not arrive and depart both in the US
\t\tflighthistory_departs_from_us <- flighthistory$departure_airport_icao_code %in% us_icao_codes$icao_code
\t\tflighthistory_arrives_in_us <- flighthistory$arrival_airport_icao_code %in% us_icao_codes$icao_code
\t\tmyRows <- which ((which(flighthistory_arrives_in_us == TRUE) & which(flighthistory_departs_from_us == TRUE)) == TRUE)
\t\tflighthistory <- flighthistory[myRows,]

\t\t# remove flights that have an invalid form of datetime - either arrival or departure date
\t\tignoreRows <- c(which (is.na (flighthistory$actual_gate_departure) == TRUE)
\t\t, which (is.na (flighthistory$actual_gate_arrival) == TRUE)
\t\t, which (is.na (flighthistory$actual_runway_arrival) == TRUE)
\t\t, which (is.na (flighthistory$actual_runway_departure) == TRUE))
\t\tignoreRows <- unique (ignoreRows)

\t\tflighthistory <- flighthistory [-ignoreRows,]

\t\t# only flights that are in air at that time
\t\tflighthistory <- flighthistory [!is.na(flighthistory$actual_gate_arrival),]
\t\tflighthistory <- flighthistory [!is.na(flighthistory$actual_runway_arrival),]


\t\t# remove flights whose gate_departure time is after runway_departure_time OR whose gate arrival is before runway arrival
\t\tignoreRows <- c(which (flighthistory$actual_gate_departure > flighthistory$actual_runway_departure), which (flighthistory$actual_runway_arrival > flighthistory$actual_gate_arrival))
\t\tignoreRows <- unique (ignoreRows)

\t\tflighthistory <- flighthistory [-ignoreRows,]

\t\t

\t\tflighthistory$cutoff_time <- as.POSIXct (cutoff_days [cutoff_days$folder_name == myDate, "selected_cutoff_time"],  tz="UTC", format = "%Y-%m-%d %H:%M:%S", origin="1970-1-1"  )
\t\t
\t\ttestflighthistory <- flighthistory [flighthistory$actual_gate_arrival >= flighthistory$cutoff_time,]

 \t\t
\t\t
\t\twrite.csv (x=testflighthistory[,myNms]  ,file=paste (cv_path, myDate, "test_flights_new.csv", sep= "/"), row.names = FALSE)


\t}
", "rawMarkdown": "", "votes": null }, { "id": "19139", "postDate": "01/09/2013 19:02:22", "content": "

is there a chance of ben responding?

", "rawMarkdown": "", "votes": null }, { "id": "19733", "postDate": "01/27/2013 02:39:05", "content": "

Hi Kiran

\r\n

can you clarify where you got days.csv and what its contents are.

\r\n

what are the test_dates.

\r\n

I really appreciate your help

\r\n

Thanks

\r\n

 

\r\n
cutoff_days <- read.csv (paste (cv_path, 'days.csv', sep='/'), header=TRUE, stringsAsFactors = FALSE)
\tcutoff_days$cutoff_time <-  as.POSIXct (cutoff_days$selected_cutoff_time, tz="UTC", format = "%Y-%m-%d %H:%M:%S", origin="1970-1-1"  )

\tfor (myDate in test_dates) {
\t\tprint (paste ("for myDate", myDate, sep=":"))

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 19139, "author_name": "rkirana", "author_url": "", "post_date": "01/09/2013 19:02:22", "content": "

is there a chance of ben responding?

", "votes": null, "replies": [] }, { "id": 19733, "author_name": "drivetheresult", "author_url": "", "post_date": "01/27/2013 02:39:05", "content": "

Hi Kiran

\r\n

can you clarify where you got days.csv and what its contents are.

\r\n

what are the test_dates.

\r\n

I really appreciate your help

\r\n

Thanks

\r\n

 

\r\n
cutoff_days <- read.csv (paste (cv_path, 'days.csv', sep='/'), header=TRUE, stringsAsFactors = FALSE)
\tcutoff_days$cutoff_time <-  as.POSIXct (cutoff_days$selected_cutoff_time, tz="UTC", format = "%Y-%m-%d %H:%M:%S", origin="1970-1-1"  )

\tfor (myDate in test_dates) {
\t\tprint (paste ("for myDate", myDate, sep=":"))

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "19121": "", "19139": "", "19733": "" }, "source": "meta" }