Skip to content

Release Notes — v1.1

Release date: March 2026

This release introduces execution hooks, weave-level lookups, and quality gates — expanding the weave lifecycle with pre/post processing capabilities.

Execution Hooks

Weaves now support pre_steps and post_steps — hook steps that run before and after thread execution within a weave.

Three hook step types are available:

  • quality_gate — Run predefined quality checks against data sources and targets. Supported checks: source_freshness, row_count_delta, row_count, table_exists, and expression.
  • sql_statement — Execute arbitrary Spark SQL. Optionally capture the scalar result into a weave-scoped variable via set_var.
  • log_message — Emit a log message with ${var.name} placeholders resolved from weave variables.

Each step has an on_failure policy: abort (stop the weave) or warn (log and continue). Pre-steps default to abort; post-steps default to warn.

pre_steps:
  - type: quality_gate
    name: check_freshness
    check: source_freshness
    source: raw.orders
    max_age: "24h"

post_steps:
  - type: log_message
    message: "Pipeline complete."
    level: info

Weave-Level Lookups

Weaves can now define named lookups — shared reference datasets that threads access via source.lookup instead of declaring a direct source type.

Lookups support two materialization strategies:

  • cache — Pre-read the data and persist it as a cached DataFrame
  • broadcast — Pre-read and apply Spark broadcast join hints
lookups:
  dim_product:
    source:
      type: delta
      alias: silver.dim_product
    materialize: true
    strategy: cache

Threads reference lookups by name:

sources:
  products:
    lookup: dim_product

Weave-Scoped Variables

Weaves can declare typed variables via the variables block. Variables are set by sql_statement hook steps and referenced in downstream config as ${var.name}.

variables:
  baseline_count:
    type: int
    default: 0

pre_steps:
  - type: sql_statement
    sql: "SELECT count(*) FROM silver.fact_orders"
    set_var: baseline_count

Supported types: string, int, long, float, double, boolean, timestamp, date.

New Exception Types

  • HookError — Raised when a hook step fails with on_failure: abort. Carries hook_name, hook_type, and phase context.
  • LookupResolutionError — Raised when a thread references a lookup name not defined in the weave's lookups map. Caught during config resolution before any data is read.

Compatibility

No breaking changes. All existing configurations continue to work without modification. The new weave fields (lookups, variables, pre_steps, post_steps) are optional and default to null.

Component Version
Python 3.11
PySpark 3.5.x
Delta Lake 3.2.x
Microsoft Fabric Runtime 1.3