Gathering¶

A modular, robust Python package and CLI toolkit for automated atmospheric and satellite data gathering, verification, caching, and downloading.

9 Data Sources

Ground-based networks (AERONET, ACTRIS, Pandonia) and satellite platforms (CDSE, EUMETSAT, EarthCARE, PACE, MODIS)
Python & Jupyter First

Type-safe Task builders with IDE autocompletion, Pipeline execution with progress bars, and rich HTML rendering
CLI & YAML

Run individual commands or batch multi-source workflows via YAML configuration files with ${ENV_VAR} expansion
Built-in Integrity

Per-format verification (MD5, SHA-256, magic headers, CRC), automatic cache repair, and corrupted file detection

Supported Data Sources¶

Source	Provider	Instruments / Products	Auth
`aeronet`	NASA AERONET	AOD, Inversions, SDA	No
`actris_ares`	ACTRIS / EARLINET	Lidar profiles, In-situ optical	No
`actris_cloudnet`	ACTRIS Cloudnet	Radar, Lidar, MWR, Classification	No
`pandonia`	Pandonia Global Network	NO₂, O₃, HCHO, SO₂, H₂O	API key
`cdse`	Copernicus Data Space	Sentinel-1/2/3/5P/6	Yes
`eumdac`	EUMETSAT Data Store	MTG FCI, MSG SEVIRI, Metop, S-3	Yes
`earthcare`	ESA EarthCARE / MAAP	ATLID, CPR, MSI, BBR	Yes
`pace`	NASA PACE	OCI, SPEXone, HARP2	Yes
`modis`	NASA MODIS	Terra MOD, Aqua MYD, Combined MCD*	Yes

Quick Start¶

PythonCLIYAML

from gathering import Task, Pipeline

# Build typed tasks with IDE autocompletion
t1 = Task.aeronet(
    station="Granada",
    product="AOD15",
    start_date="2023-07-01",
    end_date="2023-07-05",
)

t2 = Task.earthcare(
    product_types=["ATL_NOM_1B"],
    collections=["EarthCAREL1InstChecked"],
    start_time="2024-07-31T13:00:00Z",
    end_time="2024-07-31T14:00:00Z",
)

# Execute with progress bar
pipeline = Pipeline([t1, t2])
results = pipeline.run(show_progress=True)
print(results.files)

# List all available sources
gathering list

# Run a single source
gathering aeronet \
  --station Granada \
  --product AOD15 \
  --start_date 2023-07-01 \
  --end_date 2023-07-05

# Run from YAML configuration
gathering config.yaml

tasks:
  - source: aeronet
    site: Granada
    start_date: "2023-07-01"
    end_date: "2023-07-05"
    product:
      - AOD15
      - AOD20

  - source: earthcare
    collections:
      - EarthCAREL1InstChecked
    product_types:
      - ATL_NOM_1B
    start_time: "2024-07-31T13:00:00Z"
    end_time: "2024-07-31T14:00:00Z"

Architecture¶

gathering/
├── base.py         # Abstract Gatherer, caching, GATHERER_FOLDER
├── credentials.py  # Credentials manager, secret store & .env generator
├── integrity.py    # Multi-format verification (MD5/SHA-256, magic, CRC)
├── logging.py      # Unified logging with Loguru
├── pipeline.py     # Task, Pipeline, tqdm, parallel & rich HTML
├── registry.py     # Dynamic registry & plugin system
├── runner.py       # YAML & config execution engine
├── cli.py          # CLI entrypoint
└── sources/        # One module per data source
    ├── aeronet.py
    ├── actris_ares.py
    ├── actris_cloudnet.py
    ├── pandonia.py
    ├── cdse.py
    ├── eumdac.py
    ├── earthcare.py
    ├── cmr_base.py    # Shared NASA CMR search & download
    ├── pace.py
    └── modis.py

Supported on Python 3.10, 3.11, and 3.12.