Changelog¶

All notable changes to this project are documented here.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]¶

[1.0.0] - 2026-09-04¶

Added¶

Packaged Built-in Examples: Embedded all 27 YAML, Python, and env configurations directly inside the package (gathering/examples/), accessible out-of-the-box via gathering examples list, gathering examples show, and gathering examples copy.
Built-in Example Execution: Direct execution of packaged examples via gathering run example:<name> or run_from_yaml("example:<name>") without requiring local files or private repository access.
DIVA Platform Default Storage: Automatic detection of the DIVA platform environment (/workspace), preconfiguring the default download directory at /workspace/.cache/gathering/ without requiring manual workspace setup.
Per-Granule Folder Hierarchy: Multi-file satellite products (EUMETSAT MTG FCI/MSG, Copernicus CDSE Sentinel-2/3) organize extracted and raw files inside individual granule folders under the date tree (.../YYYY/MM/DD/<granule_id>/).
NASA PACE Gatherer (PaceGatherer): Full support for Ocean Color Instrument (OCI), SPEXone polarimeter, and HARP2 hyper-angular polarimeter via NASA Common Metadata Repository (CMR) API with spatial ROI, temporal filtering, and multi-threaded parallel downloads.
NASA MODIS Gatherer (ModisGatherer): Comprehensive support for Terra (MOD*), Aqua (MYD*), and Combined joint retrieval products (MCD19A2 MAIAC 1km Land AOD, MCD43 BRDF and Surface Albedo 500m / 0.05° CMG) via NASA CMR.
Shared CMR Architecture (CMRGatherer): Abstract base class in gathering.sources.cmr_base providing query normalization (CMRCacheQuery), granule filtering, and UTC date-partition aware caching (<cache_dir>/modis/<sat>/<prod>/YYYY/MM/DD/).
Copernicus CDSE Gatherer (CDSEGatherer): Sentinel-2 MSI (Level 2A/1C with cloud cover filters), Sentinel-3 OLCI & SLSTR, and Sentinel-5P TROPOMI trace gases via OData API and direct S3 bucket URIs.
EUMETSAT Data Store (EumdacGatherer): MTG FCI (with intelligent geostationary ROI swath chunk calculation), MSG SEVIRI Full-Disk, Metop IASI/AVHRR, and MTG LI/IRS.
ESA EarthCARE Gatherer (EarthCAREGatherer): ATLID Lidar, CPR Radar, and MSI Multi-Spectral Imager via ESA MAAP STAC API.
Ground-Based Networks: NASA AERONET (AOD & Almucantar inversions), Pandonia Global Network (PGN trace gases), ACTRIS ARES (in-situ optical & lidar), ACTRIS Cloudnet (cloud/aerosol profiling).
Intra-Gatherer Concurrent Downloads (max_workers): Multi-threaded parallel asset downloading within individual gatherers (ThreadPoolExecutor) alongside task-level pipeline concurrency.
Spatial ROI & AERONET Site Resolution: Full Grid model integration, auto-resolving coordinates from aeronet_locations_v3.txt, STAC bounding boxes, CMR bounding boxes, and WKT polygons.
Top-Level Package Re-Exports: Direct root imports for ModisGatherer, PaceGatherer, CDSEGatherer, EumdacGatherer, EarthCAREGatherer, AeronetGatherer, ActrisAresGatherer, ActrisCloudnetGatherer, PandoniaGatherer, Grid, Task, Pipeline, list_examples, get_example_path, copy_example, read_example from gathering.
Zero-Cost Lazy Package & CLI Loading (PEP 562): Deferred imports and proxy classes (_LazyGridMeta) ensuring CLI and module startup does not eagerly load heavy scientific dependencies (xarray, pandas, folium, jinja2, pystac), speeding up CLI startup by >60% and import gathering by ~5.6x.
File Integrity & Automated Cache Repair: Magic-byte checking (NetCDF-3, NetCDF-4/HDF5, HDF4, Gzip, Zip, Bzip2), HTML error detection (disguised 401/404/500 pages), and MD5/SHA-256 verification.
Private PyPI Registry & GitLab CI/CD: Complete CI/CD workflow with Astral uv for linting, testing, and automated publishing to https://registry.grasp-cloud.com/repository/grasp-pypi/.
MkDocs Documentation Suite & YAML Gallery: Complete documentation site in English with 22+ verified YAML configuration examples, built-in CLI commands, and no external private repository dependencies.

Changed¶

Structured Logging Migration: Full migration to Loguru structured f-strings (logger.info(f"...")) across all gatherer modules, eliminating %-style format strings and supporting thread-safe {extra[gatherer]} context injection.
Pydantic v2 Full Alignment: Upgraded parameter models to Pydantic v2 ConfigDict and @model_validator(mode="before") across all satellite and ground gatherers.
Test Suite Expansion: Full test coverage expanded to 225 tests running 100% offline in ~2s with 0 socket calls and 0 flake.

Fixed¶

Pipeline Task Annotation NameError: Fixed runtime NameError: name 'datetime' is not defined in gathering/pipeline.py:6 when resolving type hints on Task builders.
NASA CMR Caching Bug: Fixed unconditional return False override in ModisGatherer and PaceGatherer that bypassed cache checking.
Duplicate Service Entry in credentials.py: Removed duplicate cdse service dictionary entry that caused key collision.
CLI Startup Latency: Eliminated eager module evaluation during CLI argument parsing.
Inaccessible Repository Links: Removed dead GitLab URLs from documentation pages and MkDocs configuration.

[0.4.2] - 2026-08-23¶

Fixed¶

The sdist shipped an unrunnable test suite: MANIFEST.in now includes the whole suite (conftest.py), plus CHANGELOG.md and examples/.
The test suite depended on live APIs: An autouse fixture now pre-populates station validation lists, allowing offline testing with all sockets blocked.

[0.4.1] - 2026-08-23¶

Fixed¶

ACTRIS Ares query string limit: Requests for large file lists are now batched properly to avoid HTTP 400 URI length limits.

[0.4.0] - 2026-08-23¶

Added¶

Standardized cross-source parameter naming for ground-based sources (station, product, start_date, end_date).
Unified credential resolution hierarchy and CLI auth commands.