Data Loader Architecture
Data Loader Architecture
Section titled “Data Loader Architecture”The rencal.core.data_loader module defines how RenCal ingests plant, generation, and weather data.
Its main design goal is to separate:
- what the rest of the system needs (
DataLoaderinterface), from - how data is fetched and validated (
LocalDataLoaderimplementation).
This keeps calibration and simulation logic storage-agnostic while preserving strict validation.
Why this abstraction exists
Section titled “Why this abstraction exists”RenCal consumes multiple data shapes and formats:
- plant metadata (tabular),
- generation time series (tabular),
- ERA5 weather cubes (xarray/netCDF).
Without a loader abstraction, these file-format details would leak into downstream logic.
With DataLoader, consumers ask for typed datasets and do not care whether the source is local files today or another backend tomorrow.
Abstract contract: DataLoader
Section titled “Abstract contract: DataLoader”DataLoader is an abstract base class with three required methods:
load_plant_data()->PlantDatasetModelload_generation_data()->GenerationDatasetModelload_era5_data()->ERA5DatasetModel
This defines RenCal’s ingestion contract for all implementations.
Concrete implementation: LocalDataLoader
Section titled “Concrete implementation: LocalDataLoader”LocalDataLoader implements the contract against local files under DOWNLOAD_DATA_DIR.
Data sources and behavior
Section titled “Data sources and behavior”- Plant data: reads CSV, standardises ID column, returns
PlantDatasetModel. - Generation data: reads Parquet, standardises ID column, returns
GenerationDatasetModel. - ERA5 data: discovers
.ncfiles, normalises time dimension totime, casts weather variables tofloat32, concatenates/sorts, filters to standard variables, returnsERA5DatasetModel.
Reliability helpers
Section titled “Reliability helpers”The loader also includes calibrated weather helpers and manifest verification utilities:
check_historical_weather()get_historical_weather()get_prefix_histograms()verify_npy_against_manifest()
verify_npy_against_manifest() performs integrity checks (SHA-256 hash, file size, and schema sanity checks), helping ensure simulation/calibration runs consume expected artifacts.
Data loaders return dataset models
Section titled “Data loaders return dataset models”The loader layer returns validated model objects from rencal.models.base and its concrete model modules rather than raw files.
| Loader method | Return type | Backing data |
|---|---|---|
load_plant_data() |
PlantDatasetModel |
pandas.DataFrame |
load_generation_data() |
GenerationDatasetModel |
pandas.DataFrame |
load_era5_data() |
ERA5DatasetModel |
xarray.Dataset |
This creates a stable handoff boundary:
Storage/files -> DataLoader implementation -> DatasetModel objects -> Calibration/SimulationExtending to new backends
Section titled “Extending to new backends”To support new sources (for example S3 or API-backed ingestion), implement a new DataLoader subclass that returns the same three model types.
class S3DataLoader(DataLoader): def load_plant_data(self) -> PlantDatasetModel: ... def load_generation_data(self) -> GenerationDatasetModel: ... def load_era5_data(self) -> ERA5DatasetModel: ...As long as the contract is preserved, downstream code remains unchanged.

