Skip to content

Release Notes — v1.19

Release date: July 2026

This release is about trust in what the engine writes and how fast it writes it. A coordinated correctness pass fixed long-standing defects in dimension history, CDC merging, incremental loading, and the write path itself — several of them silent-data hazards. Alongside it, a performance pass removed the redundant Spark work that made wide runs slow: a thread now executes its data exactly once, metadata questions stop launching jobs, and notebook display costs nothing. A new execution: block puts concurrency, logging, and sampling under configuration control.

Correct history, correct merges

Dimension and CDC loading received the deepest fixes in this release.

SCD2 dimensions keep their history. Closed rows now keep their valid_to, overwrite-column, and previous-value contents permanently — merges no longer re-stamp history rows from the third version onward, so point-in-time queries return non-overlapping intervals. New-version detection always operates on current rows, eliminating duplicate current rows. And on_no_match_source: delete / soft_delete now actually remove or flag entities absent from the source (previously silent no-ops); on versioned dimensions both act on the current row only, and system-member exclusion now works for string surrogate keys.

Existing corrupted dimensions are not auto-repaired

Upgrading stops the degradation; it does not undo it. If a dimension already has overlapping intervals or duplicate current rows from the pre-fix behavior, rebuild it from source history, or close the duplicate current rows manually (set valid_to and clear the current flag on the older duplicates).

CDC windows merge to the latest state per key. A key changing more than once per change window now merges cleanly to its final state (previously the thread failed with a merge cardinality error, and first writes wrote duplicate keys). Change-feed windows order by commit version, explicit mappings order by the watermark column, and ties resolve delete over update over insert.

Incremental loads can no longer overwrite themselves. Combining incremental_watermark with the (defaulted) overwrite write mode now fails validation — the prior behavior silently replaced the table with each incremental slice. Declare write: {mode: append} or merge.

A write path that fails loudly

Three changes harden the moment data lands in a table.

Existence probes no longer guess. A failed target-existence probe (a transient metastore or storage error) used to read as "table absent" — which routed append and merge writes onto the table-creating overwrite branch, silently replacing existing data. The thread now fails with a clear error naming the target, the mode, and the probe failure.

Every engine commit carries a lineage stamp. Single-pass writes stamp their Delta commit with a small JSON document in userMetadata, visible in DESCRIBE HISTORY: an engine marker, the run identity, and the thread name, plus best-effort context. Row-count attribution finds the engine's own stamped commit, so concurrent writers interleaving with a run no longer degrade rows_written — and the stamp's run identity joins table history to audit columns and telemetry.

Unknown is reported as unknown. In the rare case a successful write's count truly cannot be attributed, rows_written is now None with a warning on RunResult.warnings — never a false 0. Weave and loom totals sum the known values. If automation gates on rows_written == 0, treat None as "check the table", not as an empty write.

Shared targets are guarded — with a conditional escape hatch. Two threads writing the same target now fail validation unless explicit dependencies serialize them, or every writer carries a condition: — the full-load/incremental pattern. Conditions on all writers assert mutual exclusion, and the engine enforces it at runtime: if two writers' conditions both come true, the weave aborts before the second write executes. Consumers and lookups reading such a target wait for every writer.

Each thread executes once

The performance theme of this release: stop re-computing.

A thread with no gates now executes its data exactly once — the write. Row counts arrive as byproducts of the write (query observations and commit metrics) instead of separate jobs; exports reuse one computed frame instead of re-executing the pipeline per export; the CDC change window and the SCD2 merge source each compute once; target existence resolves once per thread. Per-step pipeline statistics (resolve match rates and friends) are likewise collected as a byproduct of the write — resolve-heavy threads stop paying quadratic recompute.

Around the write, the supporting cast got cheaper too: loom-level materialized lookups stay cached across all weaves; lookups read their source exactly once and small ones broadcast explicitly; column sets resolve once per declaring scope; fact FK sentinel checks and warp nullability enforcement run as single aggregations (and sentinel warnings are now exact, not sampled); wide-frame plan mutations batch where provably safe; table existence asks the catalog; and re-display of a result in a notebook launches zero Spark jobs.

One behavior change funds part of this: output samples are now opt-in via execution.capture_samples. Default runs skip the sampling jobs (the report shows a hint); preview always samples.

The execution: block

Runtime settings now live in configuration, on the loom or weave:

execution:
  log_level: verbose
  trace: true
  max_parallel_threads: 4
  capture_samples: true

max_parallel_threads bounds intra-group thread concurrency (loom default, weave override). log_level and trace in YAML now take effect (previously silent no-ops), with documented precedence against the Context argument. Settings declared where they cannot apply produce a run-start warning. See the Execution Settings guide.

Also included

  • Aliased (as:) thread entries' conditions now actually gate the thread — they were silently never evaluated.
  • Cache targets are consumed through a registry keyed by canonical target identity; connection-declared cache targets now participate.
  • Duplicate-free lookups skip duplicate-handling work entirely; the duplicate path is unchanged.
  • Non-materialized unique_key lookups validate once per weave.
  • The observability guide documents the commit lineage stamp, and the telemetry pages now describe the exact objects the engine returns.

Upgrade notes

  • Audit incremental threads before upgrading: any thread using incremental_watermark without an explicit write: block now fails validation. Add write: {mode: append} (or merge).
  • Dormant delete configs activate: threads that declared on_no_match_source: delete/soft_delete begin acting on the next run — removed entities start disappearing or being flagged. The prior behavior was non-functional.
  • Generic CDC mappings need an ordering column: mappings declaring update_value or delete_value now require load.watermark_column. Insert-only mappings and preset: delta_cdf are unaffected.
  • Same-target writers are rejected unless serialized by explicit dependencies or condition-gated on every writer. This includes connection-declared targets, which now resolve to real paths and participate in the duplicate-writer check.
  • Table-alias existence asks the catalog ("any table exists"). On Fabric this is equivalent — lakehouse tables are Delta by platform contract. In non-Fabric dev environments, an alias bound to a non-Delta catalog table now reads as existing, which flips seed.on: first_write and existence-gated branches there.
  • rows_written is int | None on ThreadResult and ThreadTelemetry. Typed consumers comparing or summing it must handle None; custom telemetry sinks should carry the null through rather than coalescing it to zero.
  • Engine commits stamp userMetadata: if you set a session-level userMetadata, engine-stamped commits override it; unstamped engine commits (merges, seeds, warp pre-initialization) still carry the session value.
  • Sampling is opt-in: set execution.capture_samples: true to restore pre-1.19 output samples in reports.
  • No schema removals and no breaking API changes.

What's next

The next cycle turns toward v2: a reshaped telemetry contract, schema-enforced configuration, and the Delta write surface — see the roadmap for direction.