Data and Simulation Pipeline
RenCal Data and Simulation Pipeline
Section titled “RenCal Data and Simulation Pipeline”This page gives a high-level map of the RenCal workflow from external data sources to simulation outputs.
Pipeline at a glance
Section titled “Pipeline at a glance”External APIs -> DataDownloader (writes local artifacts) -> DataLoader (reads and normalizes artifacts) -> Dataset Models (validated data contracts) -> Calibration (fit technology-specific relationships) -> Simulation (sample historical weather-conditioned behavior) -> Forecast-ready streams and metricsStage 1: Data acquisition
Section titled “Stage 1: Data acquisition”RenCal downloads source data from CDS, LCCC, and Elexon, then stores normalized artifacts under the project data directory.
- ERA5 weather files (
.nc) - plant + BMU mappings (
.csv) - generation time series (
.parquet)
See Data Downloader Architecture.
Stage 2: Data loading and normalization
Section titled “Stage 2: Data loading and normalization”LocalDataLoader reads local artifacts and applies ingestion-time normalization (for example plant ID alignment and ERA5 variable selection).
Stage 3: Dataset validation boundary
Section titled “Stage 3: Dataset validation boundary”Loader methods return typed model objects:
These models enforce schema and type checks (via Pydantic + model validators), creating a fail-fast boundary before calibration and simulation.
See Dataset Models.
Stage 4: Calibration
Section titled “Stage 4: Calibration”Calibration modules fit plant/technology relationships between weather drivers and generation behavior (for wind, logistic + Weibull components).
Start here:
Stage 5: Simulation and sampling
Section titled “Stage 5: Simulation and sampling”Simulation uses calibrated/historical artifacts to sample future periods while preserving seasonal structure through bucketing.
In code terms, this stage combines:
IntermittentBucketer(draw_periodblock selection by seasonal bucket),WeatherData(historical slicing + optional average overrides),BucketedDatautilities (histograms and optimized distribution transforms),VersionedColumnsNDArrayoutput (canonical streams plus transformed versions).
Start with Weather Data Sampling. For bucket design details, see Bucketing Strategy.

