{ "id": 529296, "title": "Feature Engineering Techniques for Yield Prediction", "url": "/competitions/the-future-crop-challenge/discussion/529296", "author_name": "", "post_date": "2024-08-20T01:21:58.134547200Z", "votes": 4, "comment_count": 1, "views": 0, "content": "

In this competition, we are predicting the yield of maize and wheat based on various climate variables and soil properties. To improve the model's performance, several features were engineered from the raw data. These features include statistics of climate variables, chunk-based statistics, and additional features based on soil and geographical information. Below is the description and code for feature engineering.

\n

1. Merging Datasets

\n

The maize and wheat datasets were merged into a single dataset to streamline the analysis. This allows for unified processing and feature extraction and improves the model performance.

\n
\ntrain = pd.concat([maize_df, wheat_df], axis=).reset_index(drop=)\n
\n

2. Calculating Summary Statistics

\n

For each climate variable (tas, pr, rsds), we calculated several summary statistics over the entire time window (240 days). These statistics include the mean, median, sum, minimum, and maximum values.

\n
 ():\n    \n    pr_columns = [col  col  df.columns  col.startswith()]\n    tas_columns = [col  col  df.columns  col.startswith()]\n    rsds_columns = [col  col  df.columns  col.startswith()]\n\n    \n    mean_tas = df[tas_columns].mean(axis=).rename()\n    median_tas = df[tas_columns].median(axis=).rename()\n    sum_tas = df[tas_columns].(axis=).rename()\n    min_tas = df[tas_columns].(axis=).rename()\n    max_tas = df[tas_columns].(axis=).rename()\n\n    mean_pr = df[pr_columns].mean(axis=).rename()\n    median_pr = df[pr_columns].median(axis=).rename()\n    sum_pr = df[pr_columns].(axis=).rename()\n    min_pr = df[pr_columns].(axis=).rename()\n    max_pr = df[pr_columns].(axis=).rename()\n\n    mean_rsds = df[rsds_columns].mean(axis=).rename()\n    median_rsds = df[rsds_columns].median(axis=).rename()\n    sum_rsds = df[rsds_columns].(axis=).rename()\n    min_rsds = df[rsds_columns].(axis=).rename()\n    max_rsds = df[rsds_columns].(axis=).rename()\n\n    \n    summary_df = pd.concat([mean_tas, min_tas, max_tas, median_tas, sum_tas,\n                            mean_pr, min_pr, max_pr, median_pr, sum_pr,\n                            mean_rsds, median_rsds, sum_rsds, min_rsds, max_rsds], axis=)\n\n     summary_df\n\n\nsummary_statistics = calculate_statistics(train)\ntrain = pd.concat([train, summary_statistics], axis=)\n
\n

3. Chunk-Based Statistics

\n

To capture more localized temporal patterns, climate variables were divided into chunks (e.g., 30 days each), and similar statistics (mean, median, sum, min, max) were computed for each chunk.

\n
 ():\n    categories = [, , , , ]\n    statistics = [, , , , ]\n\n    chunk_summary_df = pd.DataFrame()\n\n     category  categories:\n        category_columns = [col  col  df.columns  col.startswith(category)]\n\n        \n         i  (, (category_columns), x):\n            chunk_columns = category_columns[i:i + x]\n            chunk_df = df[chunk_columns]\n\n            chunk_stats = {\n                : (chunk_df, stat)(axis=)\n                 stat  statistics\n            }\n\n            chunk_summary_df = pd.concat([chunk_summary_df, pd.DataFrame(chunk_stats)], axis=)\n\n     chunk_summary_df\n\n\nchunk_statistics = calculate_chunk_statistics(train, )\ntrain = pd.concat([train, chunk_statistics], axis=)\n
\n

4. Soil and Geographical Features

\n

Additional features were engineered based on soil properties and geographical location. These include interactions between soil components, encoding of categorical features (e.g., crop type, climate zone), and clustering based on geographical coordinates.

\n
 ():\n    df[] = df[] * df[]\n    df[] = (df[] // ) * \n    df[] = df[] * df[]\n    df[] = (df[] + df[] + df[]) / \n\n    df[] = df[] * df[]\n    df[] = df[] * df[]\n    df[] = df[] / df[]\n\n    \n    le_crop = LabelEncoder()\n    df[] = le_crop.fit_transform(df[])\n\n    \n    coords = df[[, , , ]]\n    kmeans = KMeans(n_clusters=, random_state=).fit(coords)\n    df[] = kmeans.labels_\n\n     df\n\n\n ():\n    group[] = np.maximum(group[] - base_temp, ).cumsum()\n    group[] = (group[] > ).cumsum()\n    group[] = (group[] < ).cumsum()\n     group\n\n\n ():\n     lat >= :\n         \n     lat >= :\n         \n     lat >= -:\n         \n    :\n         \n\ndf[] = df[].apply(assign_climate_zone)\nle_climate = LabelEncoder()\ndf[] = le_climate.fit_transform(df[])\n\n ():\n    conditions = [\n       (df[] > ),\n       (df[] <= ) & (df[] > ),\n       (df[] <= )\n        ]\n    choices = [, , ]\n    df[] = np.select(conditions, choices, default=)\n     df\n\ndf = assign_geographical_zones(df)\nle_zone = LabelEncoder()\ndf[] = le_zone.fit_transform(df[])\n
\n

5. Mean Yield by Crop

\n

To capture spatial patterns in yield, the mean yield was calculated for each crop (maize and wheat) across different locations.

\n
 ():\n    df[] = df[].astype()\n\n    maize_df = df[df[] == ]\n    maize_mean_yield = maize_df.groupby([, ])[].mean().reset_index()\n    maize_mean_yield.columns = [, , ]\n\n    wheat_df = df[df[] == ]\n    wheat_mean_yield = wheat_df.groupby([, ])[].mean().reset_index()\n    wheat_mean_yield.columns = [, , ]\n\n    merged_df = pd.merge(maize_mean_yield, wheat_mean_yield, on=[, ], how=)\n\n     merged_df\n\nmeans = compute_mean_yield_by_crop(train)\ntrain = train.merge(means, on=[, ], how=)\n
", "messages": [ { "id": "2964588", "postDate": "08/20/2024 01:21:58", "content": "

In this competition, we are predicting the yield of maize and wheat based on various climate variables and soil properties. To improve the model's performance, several features were engineered from the raw data. These features include statistics of climate variables, chunk-based statistics, and additional features based on soil and geographical information. Below is the description and code for feature engineering.

\n

1. Merging Datasets

\n

The maize and wheat datasets were merged into a single dataset to streamline the analysis. This allows for unified processing and feature extraction and improves the model performance.

\n
\ntrain = pd.concat([maize_df, wheat_df], axis=).reset_index(drop=)\n
\n

2. Calculating Summary Statistics

\n

For each climate variable (tas, pr, rsds), we calculated several summary statistics over the entire time window (240 days). These statistics include the mean, median, sum, minimum, and maximum values.

\n
 ():\n    \n    pr_columns = [col  col  df.columns  col.startswith()]\n    tas_columns = [col  col  df.columns  col.startswith()]\n    rsds_columns = [col  col  df.columns  col.startswith()]\n\n    \n    mean_tas = df[tas_columns].mean(axis=).rename()\n    median_tas = df[tas_columns].median(axis=).rename()\n    sum_tas = df[tas_columns].(axis=).rename()\n    min_tas = df[tas_columns].(axis=).rename()\n    max_tas = df[tas_columns].(axis=).rename()\n\n    mean_pr = df[pr_columns].mean(axis=).rename()\n    median_pr = df[pr_columns].median(axis=).rename()\n    sum_pr = df[pr_columns].(axis=).rename()\n    min_pr = df[pr_columns].(axis=).rename()\n    max_pr = df[pr_columns].(axis=).rename()\n\n    mean_rsds = df[rsds_columns].mean(axis=).rename()\n    median_rsds = df[rsds_columns].median(axis=).rename()\n    sum_rsds = df[rsds_columns].(axis=).rename()\n    min_rsds = df[rsds_columns].(axis=).rename()\n    max_rsds = df[rsds_columns].(axis=).rename()\n\n    \n    summary_df = pd.concat([mean_tas, min_tas, max_tas, median_tas, sum_tas,\n                            mean_pr, min_pr, max_pr, median_pr, sum_pr,\n                            mean_rsds, median_rsds, sum_rsds, min_rsds, max_rsds], axis=)\n\n     summary_df\n\n\nsummary_statistics = calculate_statistics(train)\ntrain = pd.concat([train, summary_statistics], axis=)\n
\n

3. Chunk-Based Statistics

\n

To capture more localized temporal patterns, climate variables were divided into chunks (e.g., 30 days each), and similar statistics (mean, median, sum, min, max) were computed for each chunk.

\n
 ():\n    categories = [, , , , ]\n    statistics = [, , , , ]\n\n    chunk_summary_df = pd.DataFrame()\n\n     category  categories:\n        category_columns = [col  col  df.columns  col.startswith(category)]\n\n        \n         i  (, (category_columns), x):\n            chunk_columns = category_columns[i:i + x]\n            chunk_df = df[chunk_columns]\n\n            chunk_stats = {\n                : (chunk_df, stat)(axis=)\n                 stat  statistics\n            }\n\n            chunk_summary_df = pd.concat([chunk_summary_df, pd.DataFrame(chunk_stats)], axis=)\n\n     chunk_summary_df\n\n\nchunk_statistics = calculate_chunk_statistics(train, )\ntrain = pd.concat([train, chunk_statistics], axis=)\n
\n

4. Soil and Geographical Features

\n

Additional features were engineered based on soil properties and geographical location. These include interactions between soil components, encoding of categorical features (e.g., crop type, climate zone), and clustering based on geographical coordinates.

\n
 ():\n    df[] = df[] * df[]\n    df[] = (df[] // ) * \n    df[] = df[] * df[]\n    df[] = (df[] + df[] + df[]) / \n\n    df[] = df[] * df[]\n    df[] = df[] * df[]\n    df[] = df[] / df[]\n\n    \n    le_crop = LabelEncoder()\n    df[] = le_crop.fit_transform(df[])\n\n    \n    coords = df[[, , , ]]\n    kmeans = KMeans(n_clusters=, random_state=).fit(coords)\n    df[] = kmeans.labels_\n\n     df\n\n\n ():\n    group[] = np.maximum(group[] - base_temp, ).cumsum()\n    group[] = (group[] > ).cumsum()\n    group[] = (group[] < ).cumsum()\n     group\n\n\n ():\n     lat >= :\n         \n     lat >= :\n         \n     lat >= -:\n         \n    :\n         \n\ndf[] = df[].apply(assign_climate_zone)\nle_climate = LabelEncoder()\ndf[] = le_climate.fit_transform(df[])\n\n ():\n    conditions = [\n       (df[] > ),\n       (df[] <= ) & (df[] > ),\n       (df[] <= )\n        ]\n    choices = [, , ]\n    df[] = np.select(conditions, choices, default=)\n     df\n\ndf = assign_geographical_zones(df)\nle_zone = LabelEncoder()\ndf[] = le_zone.fit_transform(df[])\n
\n

5. Mean Yield by Crop

\n

To capture spatial patterns in yield, the mean yield was calculated for each crop (maize and wheat) across different locations.

\n
 ():\n    df[] = df[].astype()\n\n    maize_df = df[df[] == ]\n    maize_mean_yield = maize_df.groupby([, ])[].mean().reset_index()\n    maize_mean_yield.columns = [, , ]\n\n    wheat_df = df[df[] == ]\n    wheat_mean_yield = wheat_df.groupby([, ])[].mean().reset_index()\n    wheat_mean_yield.columns = [, , ]\n\n    merged_df = pd.merge(maize_mean_yield, wheat_mean_yield, on=[, ], how=)\n\n     merged_df\n\nmeans = compute_mean_yield_by_crop(train)\ntrain = train.merge(means, on=[, ], how=)\n
", "rawMarkdown": "In this competition, we are predicting the yield of maize and wheat based on various climate variables and soil properties. To improve the model's performance, several features were engineered from the raw data. These features include statistics of climate variables, chunk-based statistics, and additional features based on soil and geographical information. Below is the description and code for feature engineering.\n\n### 1. Merging Datasets\nThe maize and wheat datasets were merged into a single dataset to streamline the analysis. This allows for unified processing and feature extraction and improves the model performance.\n\n```python\n# Merge maize and wheat datasets into a single dataset\ntrain = pd.concat([maize_df, wheat_df], axis=0).reset_index(drop=True)\n```\n\n### 2. Calculating Summary Statistics\nFor each climate variable (`tas`, `pr`, `rsds`), we calculated several summary statistics over the entire time window (240 days). These statistics include the mean, median, sum, minimum, and maximum values.\n\n```python\ndef calculate_statistics(df):\n # Filter columns by climate variable prefixes\n pr_columns = [col for col in df.columns if col.startswith('pr_')]\n tas_columns = [col for col in df.columns if col.startswith('tas_')]\n rsds_columns = [col for col in df.columns if col.startswith('rsds_')]\n\n # Calculate statistics for temperature (tas), precipitation (pr), and radiation (rsds)\n mean_tas = df[tas_columns].mean(axis=1).rename('mean_tas')\n median_tas = df[tas_columns].median(axis=1).rename('median_tas')\n sum_tas = df[tas_columns].sum(axis=1).rename('sum_tas')\n min_tas = df[tas_columns].min(axis=1).rename('min_tas')\n max_tas = df[tas_columns].max(axis=1).rename('max_tas')\n \n mean_pr = df[pr_columns].mean(axis=1).rename('mean_pr')\n median_pr = df[pr_columns].median(axis=1).rename('median_pr')\n sum_pr = df[pr_columns].sum(axis=1).rename('sum_pr')\n min_pr = df[pr_columns].min(axis=1).rename('min_pr')\n max_pr = df[pr_columns].max(axis=1).rename('max_pr')\n \n mean_rsds = df[rsds_columns].mean(axis=1).rename('mean_rsds')\n median_rsds = df[rsds_columns].median(axis=1).rename('median_rsds')\n sum_rsds = df[rsds_columns].sum(axis=1).rename('sum_rsds')\n min_rsds = df[rsds_columns].min(axis=1).rename('min_rsds')\n max_rsds = df[rsds_columns].max(axis=1).rename('max_rsds')\n\n # Combine all statistics into a single DataFrame\n summary_df = pd.concat([mean_tas, min_tas, max_tas, median_tas, sum_tas,\n mean_pr, min_pr, max_pr, median_pr, sum_pr,\n mean_rsds, median_rsds, sum_rsds, min_rsds, max_rsds], axis=1)\n\n return summary_df\n\n# Apply the function and merge the summary statistics with the original dataset\nsummary_statistics = calculate_statistics(train)\ntrain = pd.concat([train, summary_statistics], axis=1)\n```\n\n### 3. Chunk-Based Statistics\nTo capture more localized temporal patterns, climate variables were divided into chunks (e.g., 30 days each), and similar statistics (mean, median, sum, min, max) were computed for each chunk.\n\n```python\ndef calculate_chunk_statistics(df, x):\n categories = ['pr_', 'tas_', 'rsds_', 'tasmax_', 'tasmin_']\n statistics = ['mean', 'median', 'sum', 'max', 'min']\n \n chunk_summary_df = pd.DataFrame()\n\n for category in categories:\n category_columns = [col for col in df.columns if col.startswith(category)]\n \n # Calculate statistics for each chunk\n for i in range(0, len(category_columns), x):\n chunk_columns = category_columns[i:i + x]\n chunk_df = df[chunk_columns]\n \n chunk_stats = {\n f'{category}{stat}_chunk_{i//x}': getattr(chunk_df, stat)(axis=1)\n for stat in statistics\n }\n \n chunk_summary_df = pd.concat([chunk_summary_df, pd.DataFrame(chunk_stats)], axis=1)\n\n return chunk_summary_df\n\n# Calculate chunk-based statistics and merge with the original dataset\nchunk_statistics = calculate_chunk_statistics(train, 30)\ntrain = pd.concat([train, chunk_statistics], axis=1)\n```\n\n### 4. Soil and Geographical Features\nAdditional features were engineered based on soil properties and geographical location. These include interactions between soil components, encoding of categorical features (e.g., crop type, climate zone), and clustering based on geographical coordinates.\n\n```python\ndef generate_features(df):\n df['nitrogen_co2_interaction'] = df['soil_co2_nitrogen'] * df['soil_co2_co2']\n df['soil_co2_decade'] = (df['soil_co2_real_year'] // 10) * 10\n df['texture_nitrogen_interaction'] = df['soil_co2_texture_class'] * df['soil_co2_nitrogen']\n df['soil_health_index'] = (df['soil_co2_co2'] + df['soil_co2_nitrogen'] + df['soil_co2_texture_class']) / 3\n \n df['co2_texture_interaction'] = df['soil_co2_co2'] * df['soil_co2_texture_class']\n df['nitrogen_texture_interaction'] = df['soil_co2_nitrogen'] * df['soil_co2_texture_class']\n df['nitrogen_co2_ratio'] = df['soil_co2_nitrogen'] / df['soil_co2_co2']\n\n # Encode categorical features\n le_crop = LabelEncoder()\n df['soil_co2_crop'] = le_crop.fit_transform(df['soil_co2_crop'])\n \n # Cluster based on geographical coordinates\n coords = df[['soil_co2_lon', 'soil_co2_lat', 'climate_zone', 'geo_zone']]\n kmeans = KMeans(n_clusters=5, random_state=0).fit(coords)\n df['location_cluster'] = kmeans.labels_\n \n return df\n\n# Function to calculate GDD, heat stress days, and frost days for each group\ndef calculate_metrics(group):\n group['GDD'] = np.maximum(group['mean_tas'] - base_temp, 0).cumsum()\n group['heat_stress_days'] = (group['max_tas'] > 30).cumsum()\n group['frost_days'] = (group['min_tas'] < 0).cumsum()\n return group\n\n# Function for Geographical and Climate Zone \ndef assign_climate_zone(lat):\n if lat >= 60:\n return 'polar'\n elif lat >= 30:\n return 'temperate'\n elif lat >= -30:\n return 'tropical'\n else:\n return 'arid'\n\ndf['climate_zone'] = df['soil_co2_lat'].apply(assign_climate_zone)\nle_climate = LabelEncoder()\ndf['climate_zone'] = le_climate.fit_transform(df['climate_zone'])\n\ndef assign_geographical_zones(df):\n conditions = [\n (df['soil_co2_lat'] > 30),\n (df['soil_co2_lat'] <= 30) & (df['soil_co2_lat'] > 0),\n (df['soil_co2_lat'] <= 0)\n ]\n choices = ['North', 'Central', 'South']\n df['geo_zone'] = np.select(conditions, choices, default='Unknown')\n return df\n \ndf = assign_geographical_zones(df)\nle_zone = LabelEncoder()\ndf['geo_zone'] = le_zone.fit_transform(df['geo_zone'])\n```\n\n### 5. Mean Yield by Crop\nTo capture spatial patterns in yield, the mean yield was calculated for each crop (maize and wheat) across different locations.\n\n```python\ndef compute_mean_yield_by_crop(df):\n df['yield'] = df['yield'].astype('float64')\n\n maize_df = df[df['soil_co2_crop'] == 'maize']\n maize_mean_yield = maize_df.groupby(['soil_co2_lon', 'soil_co2_lat'])['yield'].mean().reset_index()\n maize_mean_yield.columns = ['soil_co2_lon', 'soil_co2_lat', 'maize_mean_yield']\n \n wheat_df = df[df['soil_co2_crop'] == 'wheat']\n wheat_mean_yield = wheat_df.groupby(['soil_co2_lon', 'soil_co2_lat'])['yield'].mean().reset_index()\n wheat_mean_yield.columns = ['soil_co2_lon', 'soil_co2_lat', 'wheat_mean_yield']\n\n merged_df = pd.merge(maize_mean_yield, wheat_mean_yield, on=['soil_co2_lon', 'soil_co2_lat'], how='outer')\n \n return merged_df\n\nmeans = compute_mean_yield_by_crop(train)\ntrain = train.merge(means, on=['soil_co2_lon', 'soil_co2_lat'], how='left')\n```", "votes": null }, { "id": "2964896", "postDate": "08/20/2024 10:27:18", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!", "votes": null } ], "comments": [ { "id": 2964896, "author_name": "picsoflily", "author_url": "", "post_date": "08/20/2024 10:27:18", "content": "

Thanks for sharing!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2964588": "In this competition, we are predicting the yield of maize and wheat based on various climate variables and soil properties. To improve the model's performance, several features were engineered from the raw data. These features include statistics of climate variables, chunk-based statistics, and additional features based on soil and geographical information. Below is the description and code for feature engineering.\n\n### 1. Merging Datasets\nThe maize and wheat datasets were merged into a single dataset to streamline the analysis. This allows for unified processing and feature extraction and improves the model performance.\n\n```python\n# Merge maize and wheat datasets into a single dataset\ntrain = pd.concat([maize_df, wheat_df], axis=0).reset_index(drop=True)\n```\n\n### 2. Calculating Summary Statistics\nFor each climate variable (`tas`, `pr`, `rsds`), we calculated several summary statistics over the entire time window (240 days). These statistics include the mean, median, sum, minimum, and maximum values.\n\n```python\ndef calculate_statistics(df):\n # Filter columns by climate variable prefixes\n pr_columns = [col for col in df.columns if col.startswith('pr_')]\n tas_columns = [col for col in df.columns if col.startswith('tas_')]\n rsds_columns = [col for col in df.columns if col.startswith('rsds_')]\n\n # Calculate statistics for temperature (tas), precipitation (pr), and radiation (rsds)\n mean_tas = df[tas_columns].mean(axis=1).rename('mean_tas')\n median_tas = df[tas_columns].median(axis=1).rename('median_tas')\n sum_tas = df[tas_columns].sum(axis=1).rename('sum_tas')\n min_tas = df[tas_columns].min(axis=1).rename('min_tas')\n max_tas = df[tas_columns].max(axis=1).rename('max_tas')\n \n mean_pr = df[pr_columns].mean(axis=1).rename('mean_pr')\n median_pr = df[pr_columns].median(axis=1).rename('median_pr')\n sum_pr = df[pr_columns].sum(axis=1).rename('sum_pr')\n min_pr = df[pr_columns].min(axis=1).rename('min_pr')\n max_pr = df[pr_columns].max(axis=1).rename('max_pr')\n \n mean_rsds = df[rsds_columns].mean(axis=1).rename('mean_rsds')\n median_rsds = df[rsds_columns].median(axis=1).rename('median_rsds')\n sum_rsds = df[rsds_columns].sum(axis=1).rename('sum_rsds')\n min_rsds = df[rsds_columns].min(axis=1).rename('min_rsds')\n max_rsds = df[rsds_columns].max(axis=1).rename('max_rsds')\n\n # Combine all statistics into a single DataFrame\n summary_df = pd.concat([mean_tas, min_tas, max_tas, median_tas, sum_tas,\n mean_pr, min_pr, max_pr, median_pr, sum_pr,\n mean_rsds, median_rsds, sum_rsds, min_rsds, max_rsds], axis=1)\n\n return summary_df\n\n# Apply the function and merge the summary statistics with the original dataset\nsummary_statistics = calculate_statistics(train)\ntrain = pd.concat([train, summary_statistics], axis=1)\n```\n\n### 3. Chunk-Based Statistics\nTo capture more localized temporal patterns, climate variables were divided into chunks (e.g., 30 days each), and similar statistics (mean, median, sum, min, max) were computed for each chunk.\n\n```python\ndef calculate_chunk_statistics(df, x):\n categories = ['pr_', 'tas_', 'rsds_', 'tasmax_', 'tasmin_']\n statistics = ['mean', 'median', 'sum', 'max', 'min']\n \n chunk_summary_df = pd.DataFrame()\n\n for category in categories:\n category_columns = [col for col in df.columns if col.startswith(category)]\n \n # Calculate statistics for each chunk\n for i in range(0, len(category_columns), x):\n chunk_columns = category_columns[i:i + x]\n chunk_df = df[chunk_columns]\n \n chunk_stats = {\n f'{category}{stat}_chunk_{i//x}': getattr(chunk_df, stat)(axis=1)\n for stat in statistics\n }\n \n chunk_summary_df = pd.concat([chunk_summary_df, pd.DataFrame(chunk_stats)], axis=1)\n\n return chunk_summary_df\n\n# Calculate chunk-based statistics and merge with the original dataset\nchunk_statistics = calculate_chunk_statistics(train, 30)\ntrain = pd.concat([train, chunk_statistics], axis=1)\n```\n\n### 4. Soil and Geographical Features\nAdditional features were engineered based on soil properties and geographical location. These include interactions between soil components, encoding of categorical features (e.g., crop type, climate zone), and clustering based on geographical coordinates.\n\n```python\ndef generate_features(df):\n df['nitrogen_co2_interaction'] = df['soil_co2_nitrogen'] * df['soil_co2_co2']\n df['soil_co2_decade'] = (df['soil_co2_real_year'] // 10) * 10\n df['texture_nitrogen_interaction'] = df['soil_co2_texture_class'] * df['soil_co2_nitrogen']\n df['soil_health_index'] = (df['soil_co2_co2'] + df['soil_co2_nitrogen'] + df['soil_co2_texture_class']) / 3\n \n df['co2_texture_interaction'] = df['soil_co2_co2'] * df['soil_co2_texture_class']\n df['nitrogen_texture_interaction'] = df['soil_co2_nitrogen'] * df['soil_co2_texture_class']\n df['nitrogen_co2_ratio'] = df['soil_co2_nitrogen'] / df['soil_co2_co2']\n\n # Encode categorical features\n le_crop = LabelEncoder()\n df['soil_co2_crop'] = le_crop.fit_transform(df['soil_co2_crop'])\n \n # Cluster based on geographical coordinates\n coords = df[['soil_co2_lon', 'soil_co2_lat', 'climate_zone', 'geo_zone']]\n kmeans = KMeans(n_clusters=5, random_state=0).fit(coords)\n df['location_cluster'] = kmeans.labels_\n \n return df\n\n# Function to calculate GDD, heat stress days, and frost days for each group\ndef calculate_metrics(group):\n group['GDD'] = np.maximum(group['mean_tas'] - base_temp, 0).cumsum()\n group['heat_stress_days'] = (group['max_tas'] > 30).cumsum()\n group['frost_days'] = (group['min_tas'] < 0).cumsum()\n return group\n\n# Function for Geographical and Climate Zone \ndef assign_climate_zone(lat):\n if lat >= 60:\n return 'polar'\n elif lat >= 30:\n return 'temperate'\n elif lat >= -30:\n return 'tropical'\n else:\n return 'arid'\n\ndf['climate_zone'] = df['soil_co2_lat'].apply(assign_climate_zone)\nle_climate = LabelEncoder()\ndf['climate_zone'] = le_climate.fit_transform(df['climate_zone'])\n\ndef assign_geographical_zones(df):\n conditions = [\n (df['soil_co2_lat'] > 30),\n (df['soil_co2_lat'] <= 30) & (df['soil_co2_lat'] > 0),\n (df['soil_co2_lat'] <= 0)\n ]\n choices = ['North', 'Central', 'South']\n df['geo_zone'] = np.select(conditions, choices, default='Unknown')\n return df\n \ndf = assign_geographical_zones(df)\nle_zone = LabelEncoder()\ndf['geo_zone'] = le_zone.fit_transform(df['geo_zone'])\n```\n\n### 5. Mean Yield by Crop\nTo capture spatial patterns in yield, the mean yield was calculated for each crop (maize and wheat) across different locations.\n\n```python\ndef compute_mean_yield_by_crop(df):\n df['yield'] = df['yield'].astype('float64')\n\n maize_df = df[df['soil_co2_crop'] == 'maize']\n maize_mean_yield = maize_df.groupby(['soil_co2_lon', 'soil_co2_lat'])['yield'].mean().reset_index()\n maize_mean_yield.columns = ['soil_co2_lon', 'soil_co2_lat', 'maize_mean_yield']\n \n wheat_df = df[df['soil_co2_crop'] == 'wheat']\n wheat_mean_yield = wheat_df.groupby(['soil_co2_lon', 'soil_co2_lat'])['yield'].mean().reset_index()\n wheat_mean_yield.columns = ['soil_co2_lon', 'soil_co2_lat', 'wheat_mean_yield']\n\n merged_df = pd.merge(maize_mean_yield, wheat_mean_yield, on=['soil_co2_lon', 'soil_co2_lat'], how='outer')\n \n return merged_df\n\nmeans = compute_mean_yield_by_crop(train)\ntrain = train.merge(means, on=['soil_co2_lon', 'soil_co2_lat'], how='left')\n```", "2964896": "Thanks for sharing!" }, "source": "meta" }