Skip to content

Data Loader Architecture

The rencal.core.data_loader module defines how RenCal ingests plant, generation, and weather data. Its main design goal is to separate:

  • what the rest of the system needs (DataLoader interface), from
  • how data is fetched and validated (LocalDataLoader implementation).

This keeps calibration and simulation logic storage-agnostic while preserving strict validation.

RenCal consumes multiple data shapes and formats:

  • plant metadata (tabular),
  • generation time series (tabular),
  • ERA5 weather cubes (xarray/netCDF).

Without a loader abstraction, these file-format details would leak into downstream logic. With DataLoader, consumers ask for typed datasets and do not care whether the source is local files today or another backend tomorrow.

DataLoader is an abstract base class with three required methods:

This defines RenCal’s ingestion contract for all implementations.

LocalDataLoader implements the contract against local files under DOWNLOAD_DATA_DIR.

  • Plant data: reads CSV, standardises ID column, returns PlantDatasetModel.
  • Generation data: reads Parquet, standardises ID column, returns GenerationDatasetModel.
  • ERA5 data: discovers .nc files, normalises time dimension to time, casts weather variables to float32, concatenates/sorts, filters to standard variables, returns ERA5DatasetModel.

The loader also includes calibrated weather helpers and manifest verification utilities:

verify_npy_against_manifest() performs integrity checks (SHA-256 hash, file size, and schema sanity checks), helping ensure simulation/calibration runs consume expected artifacts.

The loader layer returns validated model objects from rencal.models.base and its concrete model modules rather than raw files.

Loader method Return type Backing data
load_plant_data() PlantDatasetModel pandas.DataFrame
load_generation_data() GenerationDatasetModel pandas.DataFrame
load_era5_data() ERA5DatasetModel xarray.Dataset

This creates a stable handoff boundary:

Storage/files -> DataLoader implementation -> DatasetModel objects -> Calibration/Simulation

To support new sources (for example S3 or API-backed ingestion), implement a new DataLoader subclass that returns the same three model types.

class S3DataLoader(DataLoader):
def load_plant_data(self) -> PlantDatasetModel: ...
def load_generation_data(self) -> GenerationDatasetModel: ...
def load_era5_data(self) -> ERA5DatasetModel: ...

As long as the contract is preserved, downstream code remains unchanged.