Dataset Models
Dataset Models
Section titled “Dataset Models”RenCal uses typed dataset models in rencal.models.base and the concrete model modules to create a consistent boundary between raw inputs and downstream calibration/simulation logic.
Instead of passing around unvalidated DataFrames or xarray objects, RenCal wraps each dataset in a model with:
- schema expectations,
- datatype checks,
- time-axis validation (where relevant),
- convenience filtering and conversion methods.
Why models exist
Section titled “Why models exist”The models layer provides three important guarantees:
- Consistency: core workflows use the same interfaces regardless of source file details.
- Validation early: structural issues are detected at load time.
- Developer ergonomics: each model includes domain-specific helpers (for date ranges, plant IDs, technology filtering, etc.).
Pydantic in RenCal
Section titled “Pydantic in RenCal”RenCal’s dataset models are built on Pydantic BaseModel. In practice, this gives us a reliable model lifecycle where validation happens when model instances are created.
We lean on a few core Pydantic features:
- Typed model fields (for example,
data: pd.DataFrameordata: xr.Dataset) to make model boundaries explicit. - Field definitions via
Field(...)for model metadata and clearer intent. - Custom validators via
@field_validatoron each concrete model (PlantDatasetModel,GenerationDatasetModel,ERA5DatasetModel) to enforce schema and domain-specific checks. - Model configuration via
model_config/ config options (in our case allowing arbitrary external types like pandas/xarray objects).
How validation flows
Section titled “How validation flows”When a loader returns a model, validation executes immediately during model construction:
- Pydantic creates the model instance and calls field validators.
- The model’s validation methods (documented under
DatasetModel) run class-level checks (required_columns,required_datatypes,date_column). - If validation passes, downstream code gets a trusted model object.
- If validation fails, an exception is raised early, close to ingestion.
This “fail-fast” pattern reduces hidden data quality issues later in calibration and simulation pipelines.
Architecture overview
Section titled “Architecture overview”RenCal models are structured as a small inheritance hierarchy:
DatasetModel (abstract, Pydantic)├── PandasDatasetModel (tabular validation rules)│ ├── PlantDatasetModel│ └── GenerationDatasetModel└── XArrayDatasetModel (multidimensional validation rules) └── ERA5DatasetModelBase abstractions
Section titled “Base abstractions”DatasetModel in rencal.models.base defines shared abstract behavior and class-level schema fields:
required_columnsrequired_datatypesdate_column
It also defines required methods for:
- required-field validation,
- datatype validation,
- temporal validation,
- conversion APIs (
to_pandas(),to_xarray()).
Adds generic DataFrame validation logic:
- required column presence,
- numeric/object dtype checks,
- datetime validation for configured
date_column.
Adds xarray-specific checks:
- required variables,
- variable dtype checks,
- required time dimension/coordinate,
- non-empty, non-null, deduplicated, ascending time coordinates.
Concrete dataset models
Section titled “Concrete dataset models”Represents plant/facility metadata (location, technology, capacity).
Notable features:
- required columns such as
plant_id,latitude,longitude,technology,capacity, - technology filtering (
filter_by_technology(),get_wind_plants()), - geographic bounds and summary representation helpers.
Represents generation/settlement time-series data.
Notable features:
- required columns:
plant_id,time,quantity, - time column enforcement and filtering by date,
- filtering by individual plant IDs and compact summaries.
Represents weather data as an xarray.Dataset.
Notable features:
- required variables (
u100,v100) and dtype constraints, - time dimension validation,
- helpers for wind/solar variable access, spatial bounds,
filter_by_date_range(), and variable sub-selection.
Relationship with the Data Loader layer
Section titled “Relationship with the Data Loader layer”The data loader contract returns these model types directly:
| Loader method | Returns |
|---|---|
load_plant_data() |
PlantDatasetModel |
load_generation_data() |
GenerationDatasetModel |
load_era5_data() |
ERA5DatasetModel |
This means ingestion backends can change while preserving the same typed interface for the rest of RenCal.

