File Integrity & Checksum Verification¶

The gathering.integrity module provides standalone utilities for verifying downloaded files and maintaining a healthy cache.

Compute Checksums¶

from gathering import compute_md5, compute_sha256

# Compute hashes in 64KB chunks (memory efficient)
md5_hash = compute_md5("path/to/file.nc")
sha256_hash = compute_sha256("path/to/file.nc")

Verify Against Expected Checksum¶

from gathering import verify_checksum

is_valid = verify_checksum(
    "path/to/file.nc",
    expected_hash="d41d8cd98f00b204e9800998ecf8427e",
    algorithm="md5",  # or "sha256"
)

Comprehensive Integrity Inspection¶

verify_file_integrity() performs multiple checks in a single call:

from gathering import verify_file_integrity

is_ok, err_msg = verify_file_integrity("path/to/file.nc", min_size=1024)

if not is_ok:
    print(f"File failed verification: {err_msg}")

Checks Performed¶

Check	Description
Existence	File exists on disk
Non-zero size	File is not 0 bytes (interrupted download)
Minimum size	File exceeds optional `min_size` threshold
NetCDF magic	Checks for `CDF` or `\x89HDF` binary header
HDF5 magic	Checks for `\x89HDF` header (EarthCARE, MODIS)
Archive CRC	ZIP container `testzip()` integrity
BZ2 validity	BZ2 decompression header check (`BZh`)
HTML rejection	Detects HTML error pages served instead of data

Per-Source Integrity¶

Each gatherer applies source-specific verification automatically during download:

Source	Verification Method
`actris_cloudnet`	API MD5 checksums + NetCDF magic headers
`actris_ares`	ZIP container CRC + extracted NetCDF validation
`earthcare`	HDF5/NetCDF-4 binary header (`\x89HDF`) + truncation detection
`pandonia`	BZ2 decompression checks + HTML error rejection + endpoint fallback
`aeronet`	AERONET header validation + HTML error page rejection
`cdse`	File size verification + NetCDF/ZIP integrity
`eumdac`	Archive integrity + extraction validation
`pace`	NetCDF-4 header validation + size checks
`modis`	HDF-EOS header validation + size checks

Cache Cleaning & Auto-Repair¶

Corrupted or incomplete files are automatically detected and cleaned:

from gathering import validate_and_clean_cache

# Inspect and optionally remove corrupted files
validate_and_clean_cache("path/to/cached_file.nc", remove_corrupt=True)

Automatic Behavior¶

During normal download operations, the gatherer:

Downloads the file to a temporary path
Runs integrity checks (magic headers, checksums, size)
Moves to the final cache path only on success
On failure: logs the error, removes the partial file, and retries

This ensures the cache never contains partially downloaded or corrupted files.

Warning

If a download is interrupted (Ctrl+C, network failure), the temporary file is cleaned up. The cache only contains verified files.