Configuration¶
Default Platform Cache Directory¶
On the DIVA platform, the download directory is already configured by default:
All downloaded observation files, satellite granules, ground-based profiles, and auxiliary assets (such as the AERONET station database) are automatically cached and indexed in /workspace/.cache/gathering/. Users do not need to configure download paths manually.
If needed for custom local development outside the platform, the root directory can optionally be overridden via the GATHERER_FOLDER environment variable (which defaults to /workspace on the DIVA platform, or ~/.cache/gathering/ in standard local environments).
Download Directory Structure¶
Each source organizes files in a consistent hierarchy under the cache directory:
Ground-Based Sources¶
/workspace/.cache/gathering/
├── aeronet/
│ └── <station>/
│ └── <product>_<start>_<end>.csv
├── actris_ares/
│ └── <station>/
│ └── <data_type>/
│ └── <filename>.nc
├── actris_cloudnet/
│ └── <station>/
│ └── <product>/
│ └── <date>_<site>_<product>.nc
└── pandonia/
└── <station>/
└── <instrument_id>/
└── <product>_<date>.txt
Satellite Sources¶
/workspace/.cache/gathering/
├── cdse/
│ └── <instrument>/
│ └── <product_type>/
│ └── <YYYY>/<MM>/<DD>/
├── eumdac/
│ └── <instrument>/
│ └── <YYYY>/<MM>/<DD>/
├── earthcare/
│ └── <collection>/
│ └── <product_type>/
│ └── <YYYY>/<MM>/<DD>/
├── pace/
│ └── <instrument>/
│ └── <product_type>/
│ └── <YYYY>/<MM>/<DD>/
└── modis/
└── <satellite>/
└── <product_type>/
└── <YYYY>/<MM>/<DD>/
Environment Variables¶
The following environment variables are recognized:
| Variable | Description | Default |
|---|---|---|
GATHERER_FOLDER |
Root workspace directory (optional override) | /workspace (DIVA) / ~ (local) |
GATHERING_CREDENTIALS_FILE |
Custom path to credentials.yaml |
~/.config/gathering/credentials.yaml |
CDSE_USERNAME |
Copernicus Data Space username | — |
CDSE_PASSWORD |
Copernicus Data Space password | — |
EUMETSAT_CONSUMER_KEY |
EUMETSAT API consumer key | — |
EUMETSAT_CONSUMER_SECRET |
EUMETSAT API consumer secret | — |
EARTHCARE_USERNAME |
EarthCARE / MAAP username | — |
EARTHCARE_PASSWORD |
EarthCARE / MAAP password | — |
EARTHCARE_MAAP_TOKEN |
EarthCARE MAAP bearer token (optional) | — |
PANDONIA_API_KEY |
Pandonia Global Network API key | — |
EARTHDATA_USERNAME |
NASA Earthdata Login username | — |
EARTHDATA_PASSWORD |
NASA Earthdata Login password | — |
.env File Support¶
You can store environment variables in a .env file that is automatically loaded:
The .env file is searched in this order:
/workspace/.env(on DIVA platform)- Current working directory
.env $GATHERER_FOLDER/.env(if set)
You can generate it automatically using gathering auth env or copy the built-in template using gathering examples copy credentials.env.example .env:
# Copernicus Data Space Ecosystem (CDSE)
CDSE_USERNAME=your_email@domain.com
CDSE_PASSWORD=your_cdse_password
# EUMETSAT Data Access Client (EUMDAC)
EUMETSAT_CONSUMER_KEY=your_eumetsat_consumer_key
EUMETSAT_CONSUMER_SECRET=your_eumetsat_consumer_secret
# ESA EarthCARE (MAAP STAC / OADS)
EARTHCARE_USERNAME=your_earthcare_username
EARTHCARE_PASSWORD=your_earthcare_password
EARTHCARE_MAAP_TOKEN=your_optional_maap_token
# Pandonia Global Network (PGN)
PANDONIA_API_KEY=your_pandonia_api_key
# NASA Earthdata Login
EARTHDATA_USERNAME=your_earthdata_username
EARTHDATA_PASSWORD=your_earthdata_password
Cache Behavior¶
By default, files already present in the cache are skipped. You can override this:
