Skip to content

rencal.core.data_loader

rencal.core.data_loader defines dataset loading interfaces and local implementations for plant, generation, ERA5, and calibrated weather artifacts.

For conceptual guidance, see:

from rencal.core.data_loader import LocalDataLoader
loader = LocalDataLoader()
plant = loader.load_plant_data()
generation = loader.load_generation_data()
era5 = loader.load_era5_data()
class DataLoader(ABC): DataLoader

Abstract loading contract for plant, generation, and ERA5 datasets.

Returns DataLoader

class LocalDataLoader(data_path: str = DOWNLOAD_DATA_DIR): LocalDataLoader

Local filesystem-backed implementation.

  • data_path str

    Base path containing plant/, generation/, era5/, and calibrated/ artifacts.

Returns LocalDataLoader

load_plant_data() -> PlantDatasetModel

Load CfD plant/location/capacity data.

Returns PlantDatasetModel

load_generation_data() -> GenerationDatasetModel

Load generation/settlement time series.

Returns GenerationDatasetModel

load_era5_data() -> ERA5DatasetModel

Load ERA5 weather data using standard variables.

Returns ERA5DatasetModel

load_plant_data(id_column: str = PLANT_ID_COLUMN) -> PlantDatasetModel

Loads plant data from the configured local plant file and normalizes the plant ID column.

  • id_column str

    Source column name containing plant identifiers.

Returns PlantDatasetModel

Raises

  • FileNotFoundError: If plant data file is missing.
load_generation_data(id_column: str = PLANT_ID_COLUMN) -> GenerationDatasetModel

Loads generation data from the configured local generation file and normalizes the plant ID column.

  • id_column str

    Source column name containing plant identifiers in the generation dataset.

Returns GenerationDatasetModel

Raises

  • FileNotFoundError: If generation data file is missing.
load_era5_data() -> ERA5DatasetModel

Loads ERA5 NetCDF files from the configured local ERA5 directory, combines them on the time dimension, and filters to standard wind/solar variables.

Returns ERA5DatasetModel

Raises

  • FileNotFoundError: If no ERA5 files are found.
  • ValueError: If no files can be loaded successfully, no valid time dimension is present, concatenation fails, or no standard ERA5 variables are available.

LocalDataLoader.path_resolver_weather_data

Section titled “LocalDataLoader.path_resolver_weather_data”
path_resolver_weather_data(basename: str) -> str | Path

Resolves calibrated weather artifact basename to local path.

  • basename str

    Name of the calibrated weather artifact (for example, a .npy filename).

Returns str | Path

check_historical_weather() -> dict[str, Any]

Returns the manifest metadata for the calibrated historical wind NPY file and validates it against its manifest (SHA-256 hash, shape, and file size).

Returns dict[str, Any] — Manifest dictionary containing artifact metadata (hash, columns, size) plus a basename key for use in downstream executors.

Raises

  • FileNotFoundError: If the calibrated wind NPY artifact does not exist.
  • ValueError: If checksum, shape, or file size validation fails.
get_prefix_histograms() -> NDArray | None

Loads prefix histogram artifact when available.

Loads calibrated prefix histograms for fast distribution lookups (memory-mapped read-only). Returns None when histogram artifacts are not available.

Returns NDArray | None

get_historical_weather() -> NDArray | None

Loads historical weather NPY artifact when available.

Loads calibrated historical wind load-factor data (memory-mapped read-only). Returns None when the artifact does not exist or cannot be loaded.

Returns NDArray | None

LocalDataLoader.verify_npy_against_manifest

Section titled “LocalDataLoader.verify_npy_against_manifest”
@staticmethod
verify_npy_against_manifest(
npy_path: str | Path,
manifest_path: str | Path,
) -> dict[str, Any]

Validates a NPY file against its JSON manifest by checking SHA-256 hash, column count (second dimension), and file size. Raises ValueError on any mismatch.

  • npy_path str | Path

    Path to the .npy file to validate.

  • manifest_path str | Path

    Path to the corresponding .manifest.json file.

Returns dict[str, Any] — The parsed manifest dictionary.

Raises ValueError on SHA-256 hash, column count, or file size mismatch.