Skip to content

Dataset Models

RenCal uses typed dataset models in rencal.models.base and the concrete model modules to create a consistent boundary between raw inputs and downstream calibration/simulation logic.

Instead of passing around unvalidated DataFrames or xarray objects, RenCal wraps each dataset in a model with:

  • schema expectations,
  • datatype checks,
  • time-axis validation (where relevant),
  • convenience filtering and conversion methods.

The models layer provides three important guarantees:

  1. Consistency: core workflows use the same interfaces regardless of source file details.
  2. Validation early: structural issues are detected at load time.
  3. Developer ergonomics: each model includes domain-specific helpers (for date ranges, plant IDs, technology filtering, etc.).

RenCal’s dataset models are built on Pydantic BaseModel. In practice, this gives us a reliable model lifecycle where validation happens when model instances are created.

We lean on a few core Pydantic features:

  • Typed model fields (for example, data: pd.DataFrame or data: xr.Dataset) to make model boundaries explicit.
  • Field definitions via Field(...) for model metadata and clearer intent.
  • Custom validators via @field_validator on each concrete model (PlantDatasetModel, GenerationDatasetModel, ERA5DatasetModel) to enforce schema and domain-specific checks.
  • Model configuration via model_config / config options (in our case allowing arbitrary external types like pandas/xarray objects).

When a loader returns a model, validation executes immediately during model construction:

  1. Pydantic creates the model instance and calls field validators.
  2. The model’s validation methods (documented under DatasetModel) run class-level checks (required_columns, required_datatypes, date_column).
  3. If validation passes, downstream code gets a trusted model object.
  4. If validation fails, an exception is raised early, close to ingestion.

This “fail-fast” pattern reduces hidden data quality issues later in calibration and simulation pipelines.

RenCal models are structured as a small inheritance hierarchy:

DatasetModel (abstract, Pydantic)
├── PandasDatasetModel (tabular validation rules)
│ ├── PlantDatasetModel
│ └── GenerationDatasetModel
└── XArrayDatasetModel (multidimensional validation rules)
└── ERA5DatasetModel

DatasetModel in rencal.models.base defines shared abstract behavior and class-level schema fields:

  • required_columns
  • required_datatypes
  • date_column

It also defines required methods for:

  • required-field validation,
  • datatype validation,
  • temporal validation,
  • conversion APIs (to_pandas(), to_xarray()).

Adds generic DataFrame validation logic:

  • required column presence,
  • numeric/object dtype checks,
  • datetime validation for configured date_column.

Adds xarray-specific checks:

  • required variables,
  • variable dtype checks,
  • required time dimension/coordinate,
  • non-empty, non-null, deduplicated, ascending time coordinates.

Represents plant/facility metadata (location, technology, capacity).

Notable features:

  • required columns such as plant_id, latitude, longitude, technology, capacity,
  • technology filtering (filter_by_technology(), get_wind_plants()),
  • geographic bounds and summary representation helpers.

Represents generation/settlement time-series data.

Notable features:

  • required columns: plant_id, time, quantity,
  • time column enforcement and filtering by date,
  • filtering by individual plant IDs and compact summaries.

Represents weather data as an xarray.Dataset.

Notable features:

  • required variables (u100, v100) and dtype constraints,
  • time dimension validation,
  • helpers for wind/solar variable access, spatial bounds, filter_by_date_range(), and variable sub-selection.

The data loader contract returns these model types directly:

Loader method Returns
load_plant_data() PlantDatasetModel
load_generation_data() GenerationDatasetModel
load_era5_data() ERA5DatasetModel

This means ingestion backends can change while preserving the same typed interface for the rest of RenCal.