Multi-Source Campaign Pipeline¶
This configuration demonstrates orchestrating 8 different observation platforms (ground networks + geostationary & polar satellites) in a single unified workflow.
Built-in example: multi_source_campaign_pipeline.yaml
Run directly: gathering run example:multi_source_campaign_pipeline.yaml | Copy locally: gathering examples copy multi_source_campaign_pipeline.yaml .
Configuration¶
# ==============================================================================
# Comprehensive Multi-Source Gathering Campaign Pipeline
# ==============================================================================
# Orchestrates Ground-Truth (AERONET, Pandonia, ACTRIS) and Satellite overpasses
# (EUMETSAT MTG FCI, MSG SEVIRI, Sentinel-3, EarthCARE) in a unified workflow.
#
# Credentials Configuration:
# Set credentials via environment variables, .env, or credentials.yaml:
# - EUMETSAT_CONSUMER_KEY & EUMETSAT_CONSUMER_SECRET
# - CDSE_USERNAME & CDSE_PASSWORD
# - EARTHCARE_USERNAME & EARTHCARE_PASSWORD (or EARTHCARE_MAAP_TOKEN)
# - PANDONIA_API_KEY
# See credentials templates or run: gathering examples show credentials.env.example
#
# Execute via CLI:
# gathering run example:multi_source_campaign_pipeline.yaml
# Or in Python:
# from gathering import run_from_yaml
# results = run_from_yaml("example:multi_source_campaign_pipeline.yaml")
# ==============================================================================
logging:
level: INFO
log_file: "gathering_campaign.log"
tasks:
# ----------------------------------------------------------------------------
# 1. Ground Truth: NASA AERONET Photometer (AOD & Inversions)
# ----------------------------------------------------------------------------
- source: aeronet
site: "Granada"
start_date: "2024-09-01"
end_date: "2024-09-02"
product:
- AOD15
- ALM15
# ----------------------------------------------------------------------------
# 2. Ground Truth: Pandonia Global Network (PGN Trace Gases)
# ----------------------------------------------------------------------------
- source: pandonia
# api_key: "your_pandonia_api_key"
station: "Granada"
start_date: "2024-09-01"
end_date: "2024-09-02"
product:
- fnvh3
- fzo3
# ----------------------------------------------------------------------------
# 3. Ground Truth: ACTRIS ARES In-situ Aerosol Optical Properties
# ----------------------------------------------------------------------------
- source: actris_ares
station: "gra"
data_type: "aer_opt"
matrix: "pm10"
start_date: "2024-09-01"
end_date: "2024-09-02"
# ----------------------------------------------------------------------------
# 4. Satellite: EUMETSAT MTG FCI (Smart ROI Chunk Selection)
# ----------------------------------------------------------------------------
- source: eumdac
# consumer_key: "your_eumetsat_consumer_key"
# consumer_secret: "your_eumetsat_consumer_secret"
instrument: FCI
start_time: "2024-09-01T10:00:00Z"
end_time: "2024-09-01T14:00:00Z"
max_workers: 4
download_minutes:
- 0
grid:
center_lat_lon: [37.164, -3.605] # Granada
size_yx: [50, 50]
resolution_m: 1000.0
projection: "latlon"
extract_entries: true
# ----------------------------------------------------------------------------
# 5. Satellite: Copernicus Data Space (CDSE) Sentinel-2 MSI
# ----------------------------------------------------------------------------
- source: cdse
# username: "your_cdse_username"
# password: "your_cdse_password"
instrument: MSI
product_type: S2MSI2A
start_time: "2024-09-01T00:00:00Z"
end_time: "2024-09-02T23:59:59Z"
max_cloud_cover: 20.0
max_workers: 4
grid:
site: "Granada"
size_yx: [50, 50]
resolution_m: 1000.0
projection: "latlon"
# ----------------------------------------------------------------------------
# 6. Satellite: NASA MODIS Terra Aerosol (MOD04_L2)
# ----------------------------------------------------------------------------
- source: modis
# username: "your_earthdata_username"
# password: "your_earthdata_password"
satellite: Terra
product_type: MOD04_L2
start_time: "2024-09-01T10:00:00Z"
end_time: "2024-09-01T14:00:00Z"
max_workers: 4
grid:
site: "Granada"
size_yx: [50, 50]
resolution_m: 10000.0
# ----------------------------------------------------------------------------
# 7. Satellite: NASA PACE OCI Hyperspectral L1C
# ----------------------------------------------------------------------------
- source: pace
# username: "your_earthdata_username"
# password: "your_earthdata_password"
instrument: OCI
product_type: L1C
start_time: "2024-09-01T10:00:00Z"
end_time: "2024-09-01T14:00:00Z"
max_workers: 4
grid:
site: "Granada"
size_yx: [50, 50]
resolution_m: 5000.0
# ----------------------------------------------------------------------------
# 8. Satellite: ESA EarthCARE Atmospheric Lidar (ATLID Level 2A)
# ----------------------------------------------------------------------------
- source: earthcare
# username: "your_earthcare_username"
# password: "your_earthcare_password"
# maap_token: "your_optional_maap_token"
product_types:
- "ATL_AER_2A"
- "ATL_TC__2A"
start_time: "2024-09-01T00:00:00Z"
end_time: "2024-09-02T23:59:59Z"
max_workers: 2
grid:
center_lat_lon: [37.164, -3.605]
size_yx: [50, 50]
resolution_m: 1000.0
projection: "latlon"
Execution¶
Execute the multi-source pipeline using the CLI:
# Run the built-in example directly:
gathering run example:multi_source_campaign_pipeline.yaml
# Or export it locally for customization and execute:
gathering examples copy multi_source_campaign_pipeline.yaml .
gathering run multi_source_campaign_pipeline.yaml
Or execute directly from Python / Jupyter:
