Hi Kagglers,
\nI've been working on this dataset for a couple of days and I have written functions to read training data and do some transformation like fft/cwt then PCA but it takes too much time. Currently, it takes about 10 hours for this process to complete (approx 8 sec/iter for a single training file).
\nIf you are working on this data set what are your approach/suggestions to make this process much faster? Or Am I taking the wrong avenue?
Hi Kagglers,
\nI've been working on this dataset for a couple of days and I have written functions to read training data and do some transformation like fft/cwt then PCA but it takes too much time. Currently, it takes about 10 hours for this process to complete (approx 8 sec/iter for a single training file).
\nIf you are working on this data set what are your approach/suggestions to make this process much faster? Or Am I taking the wrong avenue?
import dataset into database and run the feature enginining at database. It could speed up the data preparing process, especially the featuring engineering repetitively.
\nIn database, the feature engineering could be done in many mode, many time and save the results in structured tempoaray place(materialized view)
\nMy approach:
\nStage A( ETL mode by manual or airflow)
Considering the Stage A need plenty of time and repetitive work, I think it deserved to let it run at database.
\nFor example, create an function to process the sensors parameters like mean, std, skew, kurt, missing_count can be get in two steps lik following mode:
\nCREATE OR REPLACE FUNCTION public.py_thdf_stat_min_max(hour_id integer, shift integer)
\n RETURNS real[]
\n LANGUAGE plpython3u
\n COST 500
\nAS $function$
\n import pandas as pd
cols = ['sensor_1','sensor_2','sensor_3', 'sensor_4','sensor_5',\n 'sensor_6','sensor_7','sensor_8','sensor_9','sensor_10'\n ]\n\nagg_cols = []\nresult_sql = plpy.execute(f\"\"\"\n select tm.* \n from \n t_measures tm\n inner join \n v_train_per_hour vtph \n on \n tm.segment_id = vtph.segment_id \n and \n vtph.hour_to_eruption={hour_id} \n order by \n ts_id\"\"\")\n\ndf = pd.DataFrame.from_records(result_sql)[cols]\n\nif shift == 0:\n agg_cols = ['size','count','mean','median','std','skew','kurt','min','max']\nelse:\n agg_cols = ['mean','median','std','skew','kurt','min','max']\n df = df.fillna(0).diff(periods=shift)\n\ndf_stat = df.agg(agg_cols)\ndf_stat['hour_id'] = hour_id\ndf_stat['shift'] = shift\n\nreturn df_stat.values.tolist()\n$function$
\n;
create an materialized view to get data.
\nNote: In order to speed up the process, the data has been stored in array data type.
\nCREATE MATERIALIZED VIEW public.mv_train_tsdf_stat
\nTABLESPACE pg_default
\nAS SELECT t.segment_id,
\nt.time_to_eruption,
\nt.time_to_eruption / 100 / 3600 AS hour_to_eruption,
\npy_tdf_stat(t.segment_id, 0) AS df_stat,
\npy_tdf_stat(t.segment_id, 1) AS df_stat_diff_1,
\npy_tdf_stat(t.segment_id, 100) AS df_stat_diff_100
\nFROM train t
\nWITH DATA;
create view to generate the x_train for next step;
\nCREATE MATERIALIZED VIEW public.x_train
\nTABLESPACE pg_default
\nAS WITH seg2hour AS (
\n SELECT vsl.segment_id,
\n vsl.time_to_eruption,
\n vhl.hour_to_eruption,
\n vsl.nlag,
\n unnest(((((((((vsl.sensor1_stat[2:] || vsl.sensor2_stat[2:]) || vsl.sensor3_stat[2:]) || vsl.sensor4_stat[2:]) || vsl.sensor5_stat[2:]) || vsl.sensor6_stat[2:]) || vsl.sensor7_stat[2:]) || vsl.sensor8_stat[2:]) || vsl.sensor9_stat[2:]) || vsl.sensor10_stat[2:]) AS seg_stat,
\n unnest(((((((((vhl.sensor1_stat[2:] || vhl.sensor2_stat[2:]) || vhl.sensor3_stat[2:]) || vhl.sensor4_stat[2:]) || vhl.sensor5_stat[2:]) || vhl.sensor6_stat[2:]) || vhl.sensor7_stat[2:]) || vhl.sensor8_stat[2:]) || vhl.sensor9_stat[2:]) || vhl.sensor10_stat[2:]) AS hour_stat
\n FROM v_train_stat_per_sensor_lags vsl
\n JOIN v_train_stat_per_hour_lags vhl ON vhl.nlag = vsl.nlag
\n )
\nSELECT seg2hour.segment_id,
\nseg2hour.time_to_eruption,
\nseg2hour.hour_to_eruption,
\ncount(*) AS count,
\ncorr(COALESCE(NULLIF(seg2hour.seg_stat, 'NaN'::real), 0::real)::double precision, COALESCE(NULLIF(seg2hour.hour_stat, 'NaN'::real), 0::real)::double precision) AS corr
\nFROM seg2hour
\nGROUP BY seg2hour.segment_id, seg2hour.time_to_eruption, seg2hour.hour_to_eruption
\nORDER BY seg2hour.segment_id, seg2hour.hour_to_eruption
\nWITH DATA;
To sumup, one step to import dataset in to postgresql database, and do the feature engineering repetitively work in datbase. It can significantly save featuring engineering time. And eay, because it is just SQL language.
\nOnly let small piece of data runing into training/predicting.
\nThat all of my sharing.
\nIf you find it useful, pls vote it up.
\nWangYong
", "rawMarkdown": "import dataset into database and run the feature enginining at database. It could speed up the data preparing process, especially the featuring engineering repetitively. \n\nIn database, the feature engineering could be done in many mode, many time and save the results in structured tempoaray place(materialized view)\n\nMy approach:\nStage A( ETL mode by manual or airflow)\n1. Import data into data base (around 2 hour)Database: Postgres 12\n2. install database development Plugin: Py/Python(install plpython3u extension). It can let data do science computation at database with ETL data to jupyternootbook. \n3. create function to working necessary feature engineering work.\n4. create view of x_train, x_test as view\nStage B( machine learning part)\n5. using pandas to the prepared data from database.\n6. feed into lightgbm or optuna for train and hyperparameter optimization\n7. submit it if you are OK with the local CV results.\n\nConsidering the Stage A need plenty of time and repetitive work, I think it deserved to let it run at database.\n\nFor example, create an function to process the sensors parameters like mean, std, skew, kurt, missing_count can be get in two steps lik following mode:\n\n1. Create an function in database(PG12): \n\nCREATE OR REPLACE FUNCTION public.py_thdf_stat_min_max(hour_id integer, shift integer)\n RETURNS real[]\n LANGUAGE plpython3u\n COST 500\nAS $function$\n import pandas as pd\n \n cols = ['sensor_1','sensor_2','sensor_3', 'sensor_4','sensor_5',\n \t\t 'sensor_6','sensor_7','sensor_8','sensor_9','sensor_10'\n \t\t]\n \n agg_cols = []\n result_sql = plpy.execute(f\"\"\"\n\t\t\t\t\t\t\t select tm.* \n\t\t\t\t\t\t\t from \n\t\t \t\t\t\t\t t_measures tm\n\t\t\t\t\t\t\t inner join \n\t\t\t\t\t\t\t v_train_per_hour vtph \n\t\t\t\t\t\t\t on \n\t\t\t\t\t\t\t tm.segment_id = vtph.segment_id \n\t\t\t\t\t\t\t and \n\t\t\t\t\t\t\t vtph.hour_to_eruption={hour_id} \n\t\t\t\t\t\t\t order by \n\t\t\t\t\t\t\t ts_id\"\"\")\n \n df = pd.DataFrame.from_records(result_sql)[cols]\n \n if shift == 0:\n \tagg_cols = ['size','count','mean','median','std','skew','kurt','min','max']\n else:\n agg_cols = ['mean','median','std','skew','kurt','min','max']\n df = df.fillna(0).diff(periods=shift)\n \n df_stat = df.agg(agg_cols)\n df_stat['hour_id'] = hour_id\n df_stat['shift'] = shift\n \n return df_stat.values.tolist()\n\n$function$\n;\n\n2. create an materialized view to get data. \nNote: In order to speed up the process, the data has been stored in array data type. \nCREATE MATERIALIZED VIEW public.mv_train_tsdf_stat\nTABLESPACE pg_default\nAS SELECT t.segment_id,\n t.time_to_eruption,\n t.time_to_eruption / 100 / 3600 AS hour_to_eruption,\n py_tdf_stat(t.segment_id, 0) AS df_stat,\n py_tdf_stat(t.segment_id, 1) AS df_stat_diff_1,\n py_tdf_stat(t.segment_id, 100) AS df_stat_diff_100\n FROM train t\nWITH DATA;\n\n3. create view to generate the x_train for next step;\nCREATE MATERIALIZED VIEW public.x_train\nTABLESPACE pg_default\nAS WITH seg2hour AS (\n SELECT vsl.segment_id,\n vsl.time_to_eruption,\n vhl.hour_to_eruption,\n vsl.nlag,\n unnest(((((((((vsl.sensor1_stat[2:] || vsl.sensor2_stat[2:]) || vsl.sensor3_stat[2:]) || vsl.sensor4_stat[2:]) || vsl.sensor5_stat[2:]) || vsl.sensor6_stat[2:]) || vsl.sensor7_stat[2:]) || vsl.sensor8_stat[2:]) || vsl.sensor9_stat[2:]) || vsl.sensor10_stat[2:]) AS seg_stat,\n unnest(((((((((vhl.sensor1_stat[2:] || vhl.sensor2_stat[2:]) || vhl.sensor3_stat[2:]) || vhl.sensor4_stat[2:]) || vhl.sensor5_stat[2:]) || vhl.sensor6_stat[2:]) || vhl.sensor7_stat[2:]) || vhl.sensor8_stat[2:]) || vhl.sensor9_stat[2:]) || vhl.sensor10_stat[2:]) AS hour_stat\n FROM v_train_stat_per_sensor_lags vsl\n JOIN v_train_stat_per_hour_lags vhl ON vhl.nlag = vsl.nlag\n )\n SELECT seg2hour.segment_id,\n seg2hour.time_to_eruption,\n seg2hour.hour_to_eruption,\n count(*) AS count,\n corr(COALESCE(NULLIF(seg2hour.seg_stat, 'NaN'::real), 0::real)::double precision, COALESCE(NULLIF(seg2hour.hour_stat, 'NaN'::real), 0::real)::double precision) AS corr\n FROM seg2hour\n GROUP BY seg2hour.segment_id, seg2hour.time_to_eruption, seg2hour.hour_to_eruption\n ORDER BY seg2hour.segment_id, seg2hour.hour_to_eruption\nWITH DATA;\n\nTo sumup, one step to import dataset in to postgresql database, and do the feature engineering repetitively work in datbase. It can significantly save featuring engineering time. And eay, because it is just SQL language. \n\nOnly let small piece of data runing into training/predicting. \n\nThat all of my sharing.\n\nIf you find it useful, pls vote it up.\n\nWangYong", "votes": 1 }, { "id": 1064096, "postDate": "2020-10-29T17:33:44.300Z", "content": "Hi, i will just say, accordingly to my intuition regarding the spectral analysis, to work only on one sensor at the beginning, especially if you're doing fft/cwt computations. There are plenty of notebook in this competition that can help you to choose the sensor with less NA values and more 'events'. The postulate i have, after reading some articles on the seismic activity regarding volcano, is to chase for a phenomena visible by spectral analysis, and that this phenomena is available on all sensors. May be i'm wrong about this hypothesis, to be tested with correlation matrix for instance, but at least it simplify your job at the beginning of your spectral study, reduce the time need to processed the data and help you to improve your hypothesis.. You can also split the train set to generate a new train(80 % ) &test set(20 % ) for having a rather good idea of the model capacity to fill the objective. Once you get some interesting results, apply the model to others sensors.
\nJust my 2 cents feeling.
", "rawMarkdown": "Hi, i will just say, accordingly to my intuition regarding the spectral analysis, to work only on one sensor at the beginning, especially if you're doing fft/cwt computations. There are plenty of notebook in this competition that can help you to choose the sensor with less NA values and more 'events'. The postulate i have, after reading some articles on the seismic activity regarding volcano, is to chase for a phenomena visible by spectral analysis, and that this phenomena is available on all sensors. May be i'm wrong about this hypothesis, to be tested with correlation matrix for instance, but at least it simplify your job at the beginning of your spectral study, reduce the time need to processed the data and help you to improve your hypothesis.. You can also split the train set to generate a new train(80 % ) &test set(20 % ) for having a rather good idea of the model capacity to fill the objective. Once you get some interesting results, apply the model to others sensors.\n\nJust my 2 cents feeling.", "votes": 1 }, { "id": 1063091, "postDate": "2020-10-28T13:51:50.533Z", "content": "What I did is having a separate pre-processing stage and saving the result in one or several csv files. Then I upload the csv files in a database I created and run training and inference using them. This strategy is allowed by the competition rules, if I am not mistaken.
", "rawMarkdown": "What I did is having a separate pre-processing stage and saving the result in one or several csv files. Then I upload the csv files in a database I created and run training and inference using them. This strategy is allowed by the competition rules, if I am not mistaken.", "votes": 1 }, { "id": 1075992, "postDate": "2020-11-12T06:57:23.663Z", "content": "You can use my dataset. It contains all the challenge data in a way that it is fast to read and process.
\nhttps://www.kaggle.com/itamargr/volcaniceruptionsdata
Itamar
", "rawMarkdown": "You can use my dataset. It contains all the challenge data in a way that it is fast to read and process.\nhttps://www.kaggle.com/itamargr/volcaniceruptionsdata\n\nItamar", "votes": 2 }, { "id": 1064693, "postDate": "2020-10-30T12:26:38.343Z", "content": "In my processing, the following two points have made the speed considerably faster during preprocessing.
\nIt may overlap with the contents described by other people, but I will describe it just in case.
(Excuse me my English is very poor.)
\n(1)
\nfor idx in tqdm.tqdm_notebook(range(len(train_df))):\n segment_id = train_df.iloc[idx,0]\n detail_df = pd.read_csv(f'{TRAIN_FILE_PATH}/{segment_id}.csv')\n detail_df.to_parquet(f'{segment_id}.parquet')\n(2)
\nfe_types = ['min','max','range','mean',]\n\ndef preprocessing(segments, df_len, path, interpolate=False):\n maps = {}\n\n for feat in sensors:\n for fe_type in fe_types:\n maps[f'{feat}_{fe_type}'] = []\n\n for idx in tqdm.tqdm_notebook(range(df_len)):\n segment_id = segments[idx]\n detail_df = pd.read_parquet(f'{path}/{segment_id}.parquet').astype(np.float32)\n sensor_data_len = len(detail_df)\n\n for feat in sensors:\n maps[feat+'_min'].append(detail_df[feat].min())\n maps[feat+'_max'].append(detail_df[feat].max())\n maps[feat+'_range'].append(maps[feat+'_max'][idx] - maps[feat+'_min'][idx])\n maps[feat+'_mean'].append(detail_df[feat].mean())\n return maps\n\nmaps = preprocessing(train_df.segment_id.tolist(), len(train_df), TRAIN_PARQUET_PATH)\nfor feat in maps.keys():\n train_df = pd.concat([train_df, pd.DataFrame(maps[feat], columns=[feat])], axis=1) \nThe code below use pure Python to consolidate csv files into one (20gb)
\n\n
\nimport glob
\nimport csv
\ndef consolidate_train():
\n files = list(glob.glob('train*.csv'))
\n with open('consolidated-train.csv', 'w') as csvoutput:
\n for i in files:
\n with open(i,'r') as csvinput:
\n writer = csv.writer(csvoutput, lineterminator='\\n')
\n reader = csv.reader(csvinput)
\n all = []
\n row = next(reader)
\n row.append('segment_id')
\n row.append('period')
\n all.append(row)
\n c = 0
\n for row in reader:
\n row.append(str(i[6:-4]))
\n row.append(c)
\n all.append(row)
\n c = c+1
\n print(str(i[6:-4]))
\n writer.writerows(all)
consolidate_train()
\n
\n
\ndef consolidate_train_drop_headers():
\n with open(\"consolidated-train.csv\",\"r\") as inputfile, open(\"output.csv\",\"w\",newline=\"\") as outputfile:
\n csv_in = csv.reader(inputfile)
\n csv_out = csv.writer(outputfile)
\n title = next(csv_in)
\n csv_out.writerow(title)
\n for row in csv_in:
\n if row != title:
\n csv_out.writerow(row)
consolidate_train_drop_headers()
\n
\nimport glob\nimport csv\ndef consolidate_train():\n files = list(glob.glob('train\\*.csv'))\n with open('consolidated-train.csv', 'w') as csvoutput:\n for i in files:\n with open(i,'r') as csvinput:\n writer = csv.writer(csvoutput, lineterminator='\\n')\n reader = csv.reader(csvinput)\n all = [] \n row = next(reader)\n row.append('segment_id')\n row.append('period')\n all.append(row)\n c = 0 \n for row in reader:\n row.append(str(i[6:-4])) \n row.append(c)\n all.append(row)\n c = c+1\n print(str(i[6:-4]))\n writer.writerows(all)\n\nconsolidate_train()\n<\\pre>\n\n\n\n\ndef consolidate_train_drop_headers():\n with open(\"consolidated-train.csv\",\"r\") as inputfile, open(\"output.csv\",\"w\",newline=\"\") as outputfile:\n csv_in = csv.reader(inputfile)\n csv_out = csv.writer(outputfile)\n title = next(csv_in)\n csv_out.writerow(title)\n for row in csv_in:\n if row != title:\n csv_out.writerow(row)\n\nconsolidate_train_drop_headers()\n<\\pre>",
"votes": 1
},
{
"id": 1063312,
"postDate": "2020-10-28T18:17:47.687Z",
"content": "@javagarm the problem is preparing the training set, as other libraries than deep learning do not support gpu.
",
"rawMarkdown": "@javagarm the problem is preparing the training set, as other libraries than `deep learning` do not support `gpu`. "
},
{
"id": 1063306,
"postDate": "2020-10-28T18:14:03.293Z",
"content": "@catadanna Indeed I took the same approach but somehow thought there may be a better way to speed up the preprocessing. I've tried dask and numba but no help.
",
"rawMarkdown": "@catadanna Indeed I took the same approach but somehow thought there may be a better way to speed up the preprocessing. I've tried `dask` and `numba` but no help. ",
"replies": [
{
"id": 1063310,
"postDate": "2020-10-28T18:15:59.753Z",
"content": "Good luck! I keep my option, though. I aleready have my database with csv files to train, and I add more pre-processed files to it, over and over.
",
"rawMarkdown": "Good luck! I keep my option, though. I aleready have my database with csv files to train, and I add more pre-processed files to it, over and over."
}
]
},
{
"id": 1062877,
"postDate": "2020-10-28T09:24:09.570Z",
"rawMarkdown": "",
"isDeleted": true
}
],
"comments": [
{
"id": 1074875,
"author_name": "WangYong",
"author_url": "",
"post_date": "2020-11-11T07:13:08.657000",
"content": "import dataset into database and run the feature enginining at database. It could speed up the data preparing process, especially the featuring engineering repetitively.
\nIn database, the feature engineering could be done in many mode, many time and save the results in structured tempoaray place(materialized view)
\nMy approach:
\nStage A( ETL mode by manual or airflow)
\n\n- Import data into data base (around 2 hour)Database: Postgres 12
\n- install database development Plugin: Py/Python(install plpython3u extension). It can let data do science computation at database with ETL data to jupyternootbook.
\n- create function to working necessary feature engineering work.
\n- create view of x_train, x_test as view
\nStage B( machine learning part) \n- using pandas to the prepared data from database.
\n- feed into lightgbm or optuna for train and hyperparameter optimization
\n- submit it if you are OK with the local CV results.
\n
\nConsidering the Stage A need plenty of time and repetitive work, I think it deserved to let it run at database.
\nFor example, create an function to process the sensors parameters like mean, std, skew, kurt, missing_count can be get in two steps lik following mode:
\n\n- Create an function in database(PG12):
\n
\nCREATE OR REPLACE FUNCTION public.py_thdf_stat_min_max(hour_id integer, shift integer)
\n RETURNS real[]
\n LANGUAGE plpython3u
\n COST 500
\nAS $function$
\n import pandas as pd
\ncols = ['sensor_1','sensor_2','sensor_3', 'sensor_4','sensor_5',\n 'sensor_6','sensor_7','sensor_8','sensor_9','sensor_10'\n ]\n\nagg_cols = []\nresult_sql = plpy.execute(f\"\"\"\n select tm.* \n from \n t_measures tm\n inner join \n v_train_per_hour vtph \n on \n tm.segment_id = vtph.segment_id \n and \n vtph.hour_to_eruption={hour_id} \n order by \n ts_id\"\"\")\n\ndf = pd.DataFrame.from_records(result_sql)[cols]\n\nif shift == 0:\n agg_cols = ['size','count','mean','median','std','skew','kurt','min','max']\nelse:\n agg_cols = ['mean','median','std','skew','kurt','min','max']\n df = df.fillna(0).diff(periods=shift)\n\ndf_stat = df.agg(agg_cols)\ndf_stat['hour_id'] = hour_id\ndf_stat['shift'] = shift\n\nreturn df_stat.values.tolist()\n
\n$function$
\n;
\n\ncreate an materialized view to get data.
\nNote: In order to speed up the process, the data has been stored in array data type.
\nCREATE MATERIALIZED VIEW public.mv_train_tsdf_stat
\nTABLESPACE pg_default
\nAS SELECT t.segment_id,
\nt.time_to_eruption,
\nt.time_to_eruption / 100 / 3600 AS hour_to_eruption,
\npy_tdf_stat(t.segment_id, 0) AS df_stat,
\npy_tdf_stat(t.segment_id, 1) AS df_stat_diff_1,
\npy_tdf_stat(t.segment_id, 100) AS df_stat_diff_100
\nFROM train t
\nWITH DATA;
create view to generate the x_train for next step;
\nCREATE MATERIALIZED VIEW public.x_train
\nTABLESPACE pg_default
\nAS WITH seg2hour AS (
\n SELECT vsl.segment_id,
\n vsl.time_to_eruption,
\n vhl.hour_to_eruption,
\n vsl.nlag,
\n unnest(((((((((vsl.sensor1_stat[2:] || vsl.sensor2_stat[2:]) || vsl.sensor3_stat[2:]) || vsl.sensor4_stat[2:]) || vsl.sensor5_stat[2:]) || vsl.sensor6_stat[2:]) || vsl.sensor7_stat[2:]) || vsl.sensor8_stat[2:]) || vsl.sensor9_stat[2:]) || vsl.sensor10_stat[2:]) AS seg_stat,
\n unnest(((((((((vhl.sensor1_stat[2:] || vhl.sensor2_stat[2:]) || vhl.sensor3_stat[2:]) || vhl.sensor4_stat[2:]) || vhl.sensor5_stat[2:]) || vhl.sensor6_stat[2:]) || vhl.sensor7_stat[2:]) || vhl.sensor8_stat[2:]) || vhl.sensor9_stat[2:]) || vhl.sensor10_stat[2:]) AS hour_stat
\n FROM v_train_stat_per_sensor_lags vsl
\n JOIN v_train_stat_per_hour_lags vhl ON vhl.nlag = vsl.nlag
\n )
\nSELECT seg2hour.segment_id,
\nseg2hour.time_to_eruption,
\nseg2hour.hour_to_eruption,
\ncount(*) AS count,
\ncorr(COALESCE(NULLIF(seg2hour.seg_stat, 'NaN'::real), 0::real)::double precision, COALESCE(NULLIF(seg2hour.hour_stat, 'NaN'::real), 0::real)::double precision) AS corr
\nFROM seg2hour
\nGROUP BY seg2hour.segment_id, seg2hour.time_to_eruption, seg2hour.hour_to_eruption
\nORDER BY seg2hour.segment_id, seg2hour.hour_to_eruption
\nWITH DATA;
\nTo sumup, one step to import dataset in to postgresql database, and do the feature engineering repetitively work in datbase. It can significantly save featuring engineering time. And eay, because it is just SQL language.
\nOnly let small piece of data runing into training/predicting.
\nThat all of my sharing.
\nIf you find it useful, pls vote it up.
\nWangYong
",
"votes": 1,
"replies": []
},
{
"id": 1064096,
"author_name": "JcLambert",
"author_url": "",
"post_date": "2020-10-29T17:33:44.300000",
"content": "Hi, i will just say, accordingly to my intuition regarding the spectral analysis, to work only on one sensor at the beginning, especially if you're doing fft/cwt computations. There are plenty of notebook in this competition that can help you to choose the sensor with less NA values and more 'events'. The postulate i have, after reading some articles on the seismic activity regarding volcano, is to chase for a phenomena visible by spectral analysis, and that this phenomena is available on all sensors. May be i'm wrong about this hypothesis, to be tested with correlation matrix for instance, but at least it simplify your job at the beginning of your spectral study, reduce the time need to processed the data and help you to improve your hypothesis.. You can also split the train set to generate a new train(80 % ) &test set(20 % ) for having a rather good idea of the model capacity to fill the objective. Once you get some interesting results, apply the model to others sensors.
\nJust my 2 cents feeling.
",
"votes": 1,
"replies": []
},
{
"id": 1063091,
"author_name": "Catadanna",
"author_url": "",
"post_date": "2020-10-28T13:51:50.533000",
"content": "What I did is having a separate pre-processing stage and saving the result in one or several csv files. Then I upload the csv files in a database I created and run training and inference using them. This strategy is allowed by the competition rules, if I am not mistaken.
",
"votes": 1,
"replies": []
},
{
"id": 1075992,
"author_name": "Itamar Gilad",
"author_url": "",
"post_date": "2020-11-12T06:57:23.663000",
"content": "You can use my dataset. It contains all the challenge data in a way that it is fast to read and process.
\nhttps://www.kaggle.com/itamargr/volcaniceruptionsdata
\nItamar
",
"votes": 2,
"replies": []
},
{
"id": 1064693,
"author_name": "cocoa",
"author_url": "",
"post_date": "2020-10-30T12:26:38.343000",
"content": "In my processing, the following two points have made the speed considerably faster during preprocessing.
\nIt may overlap with the contents described by other people, but I will describe it just in case.
\n(Excuse me my English is very poor.)
\n(1)
\nfor idx in tqdm.tqdm_notebook(range(len(train_df))):\n segment_id = train_df.iloc[idx,0]\n detail_df = pd.read_csv(f'{TRAIN_FILE_PATH}/{segment_id}.csv')\n detail_df.to_parquet(f'{segment_id}.parquet')\n(2)
\nfe_types = ['min','max','range','mean',]\n\ndef preprocessing(segments, df_len, path, interpolate=False):\n maps = {}\n\n for feat in sensors:\n for fe_type in fe_types:\n maps[f'{feat}_{fe_type}'] = []\n\n for idx in tqdm.tqdm_notebook(range(df_len)):\n segment_id = segments[idx]\n detail_df = pd.read_parquet(f'{path}/{segment_id}.parquet').astype(np.float32)\n sensor_data_len = len(detail_df)\n\n for feat in sensors:\n maps[feat+'_min'].append(detail_df[feat].min())\n maps[feat+'_max'].append(detail_df[feat].max())\n maps[feat+'_range'].append(maps[feat+'_max'][idx] - maps[feat+'_min'][idx])\n maps[feat+'_mean'].append(detail_df[feat].mean())\n return maps\n\nmaps = preprocessing(train_df.segment_id.tolist(), len(train_df), TRAIN_PARQUET_PATH)\nfor feat in maps.keys():\n train_df = pd.concat([train_df, pd.DataFrame(maps[feat], columns=[feat])], axis=1) \nThe code below use pure Python to consolidate csv files into one (20gb)
\n\n
\nimport glob
\nimport csv
\ndef consolidate_train():
\n files = list(glob.glob('train*.csv'))
\n with open('consolidated-train.csv', 'w') as csvoutput:
\n for i in files:
\n with open(i,'r') as csvinput:
\n writer = csv.writer(csvoutput, lineterminator='\\n')
\n reader = csv.reader(csvinput)
\n all = []
\n row = next(reader)
\n row.append('segment_id')
\n row.append('period')
\n all.append(row)
\n c = 0
\n for row in reader:
\n row.append(str(i[6:-4]))
\n row.append(c)
\n all.append(row)
\n c = c+1
\n print(str(i[6:-4]))
\n writer.writerows(all)
consolidate_train()
\n
\n
\ndef consolidate_train_drop_headers():
\n with open(\"consolidated-train.csv\",\"r\") as inputfile, open(\"output.csv\",\"w\",newline=\"\") as outputfile:
\n csv_in = csv.reader(inputfile)
\n csv_out = csv.writer(outputfile)
\n title = next(csv_in)
\n csv_out.writerow(title)
\n for row in csv_in:
\n if row != title:
\n csv_out.writerow(row)
consolidate_train_drop_headers()
\n
@javagarm the problem is preparing the training set, as other libraries than deep learning do not support gpu.
@catadanna Indeed I took the same approach but somehow thought there may be a better way to speed up the preprocessing. I've tried dask and numba but no help.
Good luck! I keep my option, though. I aleready have my database with csv files to train, and I add more pre-processed files to it, over and over.
", "votes": 0, "replies": [] } ] }, { "id": 1062877, "author_name": "", "author_url": "", "post_date": "2020-10-28T09:24:09.570000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1062857": "Hi Kagglers,\nI've been working on this dataset for a couple of days and I have written functions to read training data and do some transformation like fft/cwt then PCA but it takes too much time. Currently, it takes about 10 hours for this process to complete (approx 8 sec/iter for a single training file).\nIf you are working on this data set what are your approach/suggestions to make this process much faster? Or Am I taking the wrong avenue?", "1074875": "import dataset into database and run the feature enginining at database. It could speed up the data preparing process, especially the featuring engineering repetitively. \n\nIn database, the feature engineering could be done in many mode, many time and save the results in structured tempoaray place(materialized view)\n\nMy approach:\nStage A( ETL mode by manual or airflow)\n1. Import data into data base (around 2 hour)Database: Postgres 12\n2. install database development Plugin: Py/Python(install plpython3u extension). It can let data do science computation at database with ETL data to jupyternootbook. \n3. create function to working necessary feature engineering work.\n4. create view of x_train, x_test as view\nStage B( machine learning part)\n5. using pandas to the prepared data from database.\n6. feed into lightgbm or optuna for train and hyperparameter optimization\n7. submit it if you are OK with the local CV results.\n\nConsidering the Stage A need plenty of time and repetitive work, I think it deserved to let it run at database.\n\nFor example, create an function to process the sensors parameters like mean, std, skew, kurt, missing_count can be get in two steps lik following mode:\n\n1. Create an function in database(PG12): \n\nCREATE OR REPLACE FUNCTION public.py_thdf_stat_min_max(hour_id integer, shift integer)\n RETURNS real[]\n LANGUAGE plpython3u\n COST 500\nAS $function$\n import pandas as pd\n \n cols = ['sensor_1','sensor_2','sensor_3', 'sensor_4','sensor_5',\n \t\t 'sensor_6','sensor_7','sensor_8','sensor_9','sensor_10'\n \t\t]\n \n agg_cols = []\n result_sql = plpy.execute(f\"\"\"\n\t\t\t\t\t\t\t select tm.* \n\t\t\t\t\t\t\t from \n\t\t \t\t\t\t\t t_measures tm\n\t\t\t\t\t\t\t inner join \n\t\t\t\t\t\t\t v_train_per_hour vtph \n\t\t\t\t\t\t\t on \n\t\t\t\t\t\t\t tm.segment_id = vtph.segment_id \n\t\t\t\t\t\t\t and \n\t\t\t\t\t\t\t vtph.hour_to_eruption={hour_id} \n\t\t\t\t\t\t\t order by \n\t\t\t\t\t\t\t ts_id\"\"\")\n \n df = pd.DataFrame.from_records(result_sql)[cols]\n \n if shift == 0:\n \tagg_cols = ['size','count','mean','median','std','skew','kurt','min','max']\n else:\n agg_cols = ['mean','median','std','skew','kurt','min','max']\n df = df.fillna(0).diff(periods=shift)\n \n df_stat = df.agg(agg_cols)\n df_stat['hour_id'] = hour_id\n df_stat['shift'] = shift\n \n return df_stat.values.tolist()\n\n$function$\n;\n\n2. create an materialized view to get data. \nNote: In order to speed up the process, the data has been stored in array data type. \nCREATE MATERIALIZED VIEW public.mv_train_tsdf_stat\nTABLESPACE pg_default\nAS SELECT t.segment_id,\n t.time_to_eruption,\n t.time_to_eruption / 100 / 3600 AS hour_to_eruption,\n py_tdf_stat(t.segment_id, 0) AS df_stat,\n py_tdf_stat(t.segment_id, 1) AS df_stat_diff_1,\n py_tdf_stat(t.segment_id, 100) AS df_stat_diff_100\n FROM train t\nWITH DATA;\n\n3. create view to generate the x_train for next step;\nCREATE MATERIALIZED VIEW public.x_train\nTABLESPACE pg_default\nAS WITH seg2hour AS (\n SELECT vsl.segment_id,\n vsl.time_to_eruption,\n vhl.hour_to_eruption,\n vsl.nlag,\n unnest(((((((((vsl.sensor1_stat[2:] || vsl.sensor2_stat[2:]) || vsl.sensor3_stat[2:]) || vsl.sensor4_stat[2:]) || vsl.sensor5_stat[2:]) || vsl.sensor6_stat[2:]) || vsl.sensor7_stat[2:]) || vsl.sensor8_stat[2:]) || vsl.sensor9_stat[2:]) || vsl.sensor10_stat[2:]) AS seg_stat,\n unnest(((((((((vhl.sensor1_stat[2:] || vhl.sensor2_stat[2:]) || vhl.sensor3_stat[2:]) || vhl.sensor4_stat[2:]) || vhl.sensor5_stat[2:]) || vhl.sensor6_stat[2:]) || vhl.sensor7_stat[2:]) || vhl.sensor8_stat[2:]) || vhl.sensor9_stat[2:]) || vhl.sensor10_stat[2:]) AS hour_stat\n FROM v_train_stat_per_sensor_lags vsl\n JOIN v_train_stat_per_hour_lags vhl ON vhl.nlag = vsl.nlag\n )\n SELECT seg2hour.segment_id,\n seg2hour.time_to_eruption,\n seg2hour.hour_to_eruption,\n count(*) AS count,\n corr(COALESCE(NULLIF(seg2hour.seg_stat, 'NaN'::real), 0::real)::double precision, COALESCE(NULLIF(seg2hour.hour_stat, 'NaN'::real), 0::real)::double precision) AS corr\n FROM seg2hour\n GROUP BY seg2hour.segment_id, seg2hour.time_to_eruption, seg2hour.hour_to_eruption\n ORDER BY seg2hour.segment_id, seg2hour.hour_to_eruption\nWITH DATA;\n\nTo sumup, one step to import dataset in to postgresql database, and do the feature engineering repetitively work in datbase. It can significantly save featuring engineering time. And eay, because it is just SQL language. \n\nOnly let small piece of data runing into training/predicting. \n\nThat all of my sharing.\n\nIf you find it useful, pls vote it up.\n\nWangYong", "1064096": "Hi, i will just say, accordingly to my intuition regarding the spectral analysis, to work only on one sensor at the beginning, especially if you're doing fft/cwt computations. There are plenty of notebook in this competition that can help you to choose the sensor with less NA values and more 'events'. The postulate i have, after reading some articles on the seismic activity regarding volcano, is to chase for a phenomena visible by spectral analysis, and that this phenomena is available on all sensors. May be i'm wrong about this hypothesis, to be tested with correlation matrix for instance, but at least it simplify your job at the beginning of your spectral study, reduce the time need to processed the data and help you to improve your hypothesis.. You can also split the train set to generate a new train(80 % ) &test set(20 % ) for having a rather good idea of the model capacity to fill the objective. Once you get some interesting results, apply the model to others sensors.\n\nJust my 2 cents feeling.", "1063091": "What I did is having a separate pre-processing stage and saving the result in one or several csv files. Then I upload the csv files in a database I created and run training and inference using them. This strategy is allowed by the competition rules, if I am not mistaken.", "1075992": "You can use my dataset. It contains all the challenge data in a way that it is fast to read and process.\nhttps://www.kaggle.com/itamargr/volcaniceruptionsdata\n\nItamar", "1064693": "In my processing, the following two points have made the speed considerably faster during preprocessing.\nIt may overlap with the contents described by other people, but I will describe it just in case.\n- Convert the CSV file to a parquet file in advance, and use the parquet file during preprocessing. -> (1)\n- The new feature will be kept as a list and finally combined using the concat function. -> (2)\n\n(Excuse me my English is very poor.)\n\n(1)\n```Python\nfor idx in tqdm.tqdm_notebook(range(len(train_df))):\n segment_id = train_df.iloc[idx,0]\n detail_df = pd.read_csv(f'{TRAIN_FILE_PATH}/{segment_id}.csv')\n detail_df.to_parquet(f'{segment_id}.parquet')\n```\n\n(2)\n```Python\nfe_types = ['min','max','range','mean',]\n\ndef preprocessing(segments, df_len, path, interpolate=False):\n maps = {}\n\n for feat in sensors:\n for fe_type in fe_types:\n maps[f'{feat}_{fe_type}'] = []\n \n for idx in tqdm.tqdm_notebook(range(df_len)):\n segment_id = segments[idx]\n detail_df = pd.read_parquet(f'{path}/{segment_id}.parquet').astype(np.float32)\n sensor_data_len = len(detail_df)\n \n for feat in sensors:\n maps[feat+'_min'].append(detail_df[feat].min())\n maps[feat+'_max'].append(detail_df[feat].max())\n maps[feat+'_range'].append(maps[feat+'_max'][idx] - maps[feat+'_min'][idx])\n maps[feat+'_mean'].append(detail_df[feat].mean())\n return maps\n\nmaps = preprocessing(train_df.segment_id.tolist(), len(train_df), TRAIN_PARQUET_PATH)\nfor feat in maps.keys():\n train_df = pd.concat([train_df, pd.DataFrame(maps[feat], columns=[feat])], axis=1) \n```", "1063981": "The code below use pure Python to consolidate csv files into one (20gb)\n\n\nimport glob\nimport csv\ndef consolidate_train():\n files = list(glob.glob('train\\*.csv'))\n with open('consolidated-train.csv', 'w') as csvoutput:\n for i in files:\n with open(i,'r') as csvinput:\n writer = csv.writer(csvoutput, lineterminator='\\n')\n reader = csv.reader(csvinput)\n all = [] \n row = next(reader)\n row.append('segment_id')\n row.append('period')\n all.append(row)\n c = 0 \n for row in reader:\n row.append(str(i[6:-4])) \n row.append(c)\n all.append(row)\n c = c+1\n print(str(i[6:-4]))\n writer.writerows(all)\n\nconsolidate_train()\n<\\pre>\n\n\n\n\ndef consolidate_train_drop_headers():\n with open(\"consolidated-train.csv\",\"r\") as inputfile, open(\"output.csv\",\"w\",newline=\"\") as outputfile:\n csv_in = csv.reader(inputfile)\n csv_out = csv.writer(outputfile)\n title = next(csv_in)\n csv_out.writerow(title)\n for row in csv_in:\n if row != title:\n csv_out.writerow(row)\n\nconsolidate_train_drop_headers()\n<\\pre>",
"1063312": "@javagarm the problem is preparing the training set, as other libraries than `deep learning` do not support `gpu`. ",
"1063306": "@catadanna Indeed I took the same approach but somehow thought there may be a better way to speed up the preprocessing. I've tried `dask` and `numba` but no help. ",
"1062877": ""
}
}