Data Downloader Architecture
Data Downloader Architecture
Section titled “Data Downloader Architecture”The rencal.core.data_downloader module is responsible for acquiring source datasets used by RenCal calibration and simulation workflows.
Where the Data Loader Architecture focuses on reading local artifacts into validated models, the downloader layer is responsible for creating those local artifacts in the first place.
Why this module exists
Section titled “Why this module exists”RenCal depends on multiple external data providers with different protocols and formats:
- Copernicus CDS API for ERA5 weather (
.nc), - LCCC APIs/CSV for CfD register and BMU mapping (
.csv), - Elexon BMRS API for generation settlement data (JSON -> parquet).
data_downloader.py centralises this acquisition logic behind shared abstractions so ingestion remains reproducible and easier to operate.
Core abstraction: DataDownloader
Section titled “Core abstraction: DataDownloader”DataDownloader is an abstract base class that provides:
- a shared output root (
DOWNLOAD_DATA_DIR), _update_output_directory(...)for source-specific subfolders (seeDataDownloader),- a required
download()method for concrete implementations.
This creates a consistent contract across all external data download flows.
Concrete downloaders
Section titled “Concrete downloaders”Responsibilities:
- determine download years from
CALIBRATION_START_DATEandCALIBRATION_END_DATE, - authenticate with CDS (
cdsapi.Client) using env/config, - request wind and solar variables separately, then merge to a single yearly NetCDF,
- verify file structure and annotate metadata,
- skip or verify existing historical files while refreshing current-year outputs.
Notable reliability mechanisms:
- lazy CDS client initialization with explicit error messages for missing API key,
- per-year verification via
_verify_and_update_metadata(...)onERA5DataDownloader, - progress visibility via
tqdm.
Responsibilities:
- download CfD register data,
- download CfD-to-BMU mapping CSV,
- normalise key fields and merge datasets,
- persist plant-level output to
plant/{PLANT_DATA_FILE_NAME}.
This dataset is a dependency for generation downloads and for plant-level model ingestion.
Responsibilities:
- load locally saved CfD/BMU data to determine relevant BMU IDs,
- request settled generation data from Elexon over calibration window,
- map settlement periods to hourly UTC timestamps,
- handle daylight-saving edge cases (normal/short/long settlement days),
- allocate shared BMU generation proportionally to plant capacity,
- aggregate to plant-level hourly quantities and save parquet.
This stage converts operational market settlement records into model-ready time series.
DownloadManager composes the three concrete downloaders and exposes:
download_all() enforces dependency order:
- CfD register + BMU mapping,
- generation data (depends on BMUs from CfD data),
- ERA5 weather data.
Downloader vs loader boundary
Section titled “Downloader vs loader boundary”The two modules are complementary:
- Downloader layer (
rencal.core.data_downloader): fetches and writes raw/normalized artifacts. - Loader layer (
rencal.core.data_loader): reads those artifacts and returns validated dataset models.
External APIs/files -> DataDownloader -> local artifacts -> DataLoader -> DatasetModel objectsRelationship to dataset models
Section titled “Relationship to dataset models”The downloader does not directly instantiate model classes, but it produces the files consumed by the loader methods that return:
For model-layer details and validation behavior, see Dataset Models.
Operational notes
Section titled “Operational notes”- Credentials: ERA5 requires a valid
CDS_API_KEY(or explicitapi_key). - Idempotence: existing outputs are reused/skipped where appropriate.
- Temporal correctness: generation conversion logic accounts for UK settlement-day DST behavior before UTC aggregation.

