YAML Configuration¶

gathering supports defining multi-source gathering workflows using YAML configuration files with environment variable expansion.

Basic Structure¶

# Optional logging configuration
logging:
  level: INFO               # DEBUG, INFO, WARNING, ERROR
  log_file: "gathering.log" # Optional log file path

# List of gathering tasks
tasks:
  - source: <source_name>
    # Authentication (optional — resolved from env/credentials.yaml)
    # auth:
    #   username: ${ENV_VAR}
    #   password: ${ENV_VAR}

    # Fetch parameters (source-specific)
    <parameter1>: <value1>
    <parameter2>: <value2>

    # Cache control
    overwrite_cache: false

Environment Variable Expansion¶

YAML files support ${VARIABLE} syntax to reference environment variables:

tasks:
  - source: earthcare
    auth:
      username: ${EARTHCARE_USERNAME}
      password: ${EARTHCARE_PASSWORD}
    collections:
      - EARTHCARE_ATLID_L1B
    product_types:
      - ATL_NOM_1B
    start_time: "2024-09-01T00:00:00Z"
    end_time: "2024-09-02T23:59:59Z"

Variables are loaded from the environment, .env files, or credentials.yaml (see Authentication).

Single-Task Shorthand¶

For single-task files, you can omit the tasks: wrapper:

source: pandonia
station: "Innsbruck"
instrument_id: "Pandora45"
start_date: "2023-07-01"
end_date: "2023-07-03"
product:
  - "fnvh3"
  - "fzo3"
level: "L2Fit"
spectrometer: "1"
overwrite_cache: false

Multi-Task Workflow¶

Combine ground-based and satellite sources in a single file:

logging:
  level: INFO
  log_file: "gathering_campaign.log"

tasks:
  # Ground Truth: AERONET
  - source: aeronet
    site: "Granada"
    start_date: "2024-09-01"
    end_date: "2024-09-02"
    product:
      - AOD15
      - ALM15

  # Ground Truth: Pandonia
  - source: pandonia
    station: "Granada"
    start_date: "2024-09-01"
    end_date: "2024-09-02"
    product:
      - fnvh3
      - fzo3

  # Satellite: EUMETSAT MTG FCI
  - source: eumdac
    instrument: FCI
    start_time: "2024-09-01T10:00:00Z"
    end_time: "2024-09-01T14:00:00Z"
    max_workers: 4
    download_minutes:
      - 0
    grid:
      center_lat_lon: [37.164, -3.605]
      size_yx: [50, 50]
      resolution_m: 1000.0
      projection: "latlon"
    extract_entries: true

  # Satellite: EarthCARE ATLID
  - source: earthcare
    collections:
      - "EARTHCARE_ATLID_L2A"
    product_types:
      - "ATL_AER_2A"
      - "ATL_TC__2A"
    start_time: "2024-09-01T00:00:00Z"
    end_time: "2024-09-02T23:59:59Z"
    max_workers: 2
    grid:
      center_lat_lon: [37.164, -3.605]
      size_yx: [50, 50]
      resolution_m: 1000.0
      projection: "latlon"

Executing YAML Files¶

CLIPython

gathering config.yaml

# With verbose logging
gathering --verbose --log-file run.log config.yaml

from gathering import run_from_yaml

results = run_from_yaml("config.yaml")
print(results.files)

Grid / Spatial ROI in YAML¶

Satellite sources support spatial Region of Interest (ROI) via the grid block:

# Option 1: AERONET station auto-resolution
grid:
  site: "Granada"          # Resolved from aeronet_locations_v3.txt
  size_yx: [100, 100]
  resolution_m: 1000.0
  projection: "latlon"

# Option 2: Explicit center coordinates
grid:
  center_lat_lon: [37.164, -3.605]
  size_yx: [100, 100]
  resolution_m: 1000.0
  projection: "latlon"

Alternatively, some sources support bounding_box:

# [min_lon, min_lat, max_lon, max_lat]
bounding_box: [-10.0, 35.0, 5.0, 45.0]

See Spatial Grid & AERONET Sites for more details.

Complete Example Files¶

Visit the YAML Examples Gallery to consult, browse, and copy full configuration files for all supported sources:

Multi-Source Campaign Pipeline
Ground-Based Networks Examples (AERONET, Pandonia, ACTRIS ARES, ACTRIS Cloudnet)
NASA Satellites Examples (MODIS Terra/Aqua/Combined & PACE OCI/HARP2/SPEXone)
Copernicus CDSE Satellites Examples (Sentinel-2, Sentinel-3, Sentinel-5P)
EUMETSAT Data Store Satellites Examples (MTG FCI/LI/IRS, MSG SEVIRI, Metop IASI/AVHRR)
ESA EarthCARE Satellite Examples (ATLID Lidar, CPR Radar, MSI Imager)
Credentials Templates (credentials.yaml & .env)