Weather Data Sampling
Weather Data Sampling
Section titled “Weather Data Sampling”This page explains the simulation-side weather sampling concepts implemented in rencal.simulation.weather_data and rencal.simulation.intermittent_bucketer.
Problem this solves
Section titled “Problem this solves”RenCal needs future hourly weather-conditioned generation streams that are:
- statistically plausible,
- seasonally aware,
- compatible with calibrated plant behavior.
Rather than naive random draws, the simulation layer samples historical periods with bucketing and distribution controls.
Core components
Section titled “Core components”HistoricalMetadata describes a historical NPY artifact and its temporal limits:
- artifact basename,
- path resolver,
- left/right time bounds,
- column list,
- optional block size (
hours_per_block) for histogram acceleration.
This descriptor can also be built from a loader-provided manifest via HistoricalMetadata.from_manifest(...), keeping simulation metadata aligned with calibrated artifact metadata.
WeatherData.random_sample(...) returns a VersionedColumnsNDArray, not a plain array.
This wrapper keeps track of:
- canonical columns from the base sampled data,
- appended transformed versions for columns that have one or more desired-average overrides.
Use:
sample.access_col(canonical_col)for the canonical stream,sample.access_col(canonical_col, version=1)(or higher) for transformed variants.
WeatherData is the main sampler object.
It uses metadata + optional desired-average overrides and then samples historical periods for a requested future window.
Constructor semantics that matter in practice:
desired_averagesis a mapping:dict[int, list[float]], where each key is a canonical column index and each value is one or more target averages.- If desired averages are provided, at least one of
historical_dataorprefix_histogramsmust be provided at construction (otherwise optimization cannot be computed). ignore_zeros=Trueactivates solar-style behavior: histogram building and resampling operate only on non-zero hours, while zeros are preserved.
Key behaviors include:
- process-scope cache by NPY basename,
- bucket-aware random draw selection,
- optional zero handling mode (
ignore_zeros) for solar-like series, - histogram/inverse-CDF style resampling utilities,
- version-aware output for duplicated target averages per stream.
IntermittentBucketer applies seasonality-aware bucketing with a single category.
- Date windows are assigned to monthly buckets.
- Draws are selected from historical ranges that match bucket context.
- Resulting samples preserve seasonal structure better than unconstrained random shuffling.
For a deeper explanation of bucket design and draw mechanics, see Bucketing Strategy.
Sampling modes and zero handling
Section titled “Sampling modes and zero handling”WeatherData supports two conceptual modes:
ignore_zeros = False: all values contribute to histograms and resampling (typical wind behavior).ignore_zeros = True: zero-valued hours are preserved; only non-zero values are resampled (typical solar behavior where nighttime zeros are structural).
This avoids distorting physical patterns (for example, introducing solar output at night).
When ignore_zeros=True, desired averages are internally rescaled from all-hours to non-zero-hours before optimization, so transformed daytime values still target the intended whole-series behavior.
Sampling workflow (code-level)
Section titled “Sampling workflow (code-level)”WeatherData.random_sample(...) works as follows:
- load/mmap historical NPY via process-scope cache (
_HIST_CACHE), - extend right-hand horizon with
BucketedData.adjust_forecasted_horizon(...)so draw-period chunking can fully cover the request, - call the inherited
Bucketer.random_sample(...)to get historical block start dates, - slice historical arrays in
draw_period * 24hour chunks and concatenate, - trim to exactly requested hour count,
- append transformed columns (if any) via
BucketedData.append_resampled_columns(...), - return
VersionedColumnsNDArray.
Histogram path and optimization path
Section titled “Histogram path and optimization path”When desired averages are present, the constructor precomputes old/new distributions:
- Uses
BucketedData.crop_time_margins_to_full_years()to avoid seasonal bias in training windows. - Uses
BucketedData.get_fast_histogram(...)if prefix histograms andhours_per_blockare available; otherwise falls back toBucketedData.get_histogram(...). - Computes optimized distributions via
BucketedData.compute_optimized_distributions(...)(exponential tilting / KL-minimizing formulation), then storesnew_cdffor inverse-CDF mapping.
How it connects to loader/downloader layers
Section titled “How it connects to loader/downloader layers”Simulation sampling depends on upstream artifacts:
- downloader produces local weather/generation artifacts,
- loader validates calibrated weather artifacts + manifests,
- simulation reads those validated historical artifacts for draw/resample operations.
See:

