Plumbline
An evaluation and verification gate for AI-generated SQL and dbt. The public starter kit proves the free tier: does a candidate query return the right answer. The private kit this page describes adds the second axis: is it the right answer for the right reasons.
Architecture
context layer (metrics.yml, generation skill,
project-context conventions, pack_lint)
│
▼
AI agent generates governed dbt / SQL
│
▼
dbt + DuckDB warehouse
staging → intermediate → marts
│
▼
eval harness: golden set
two-axis scorer: correctness, governance
│
▼
review gate ── PASS ──▶ shipped to stakeholder
│
FAIL
▼
blocked, named failure mode reported
A versioned context pack frames the agent so it generates correct, governed SQL instead of guessing: metric definitions, model-doc standards, project conventions, and a lint step that fails if the pack is internally inconsistent or has leaked a golden answer into a file the agent reads. dbt models flow staging to intermediate to marts on a local DuckDB warehouse, staging as the only layer allowed to touch raw data, marts as the governed public API.
Validation gateEvery candidate query is scored on two axes. Correctness asks whether the result set matches a hand-verified golden answer, compared as an order-independent, numerically tolerant multiset so any correctly-shaped query passes. Governance asks whether the query is right for the right reasons: does it read the governed mart, avoid a raw-layer leak, avoid a memorized constant. A query is allowed through only when both axes hold, and a blocked query is tagged with a named failure mode and the pipeline stage it failed at, so a binary fail becomes a diagnosis.
Architecture & Trade-offs
What was built, and what was deliberately chosen against.
Semantic equivalence, not string match
An analytics agent legitimately produces many correct query shapes for one answer. Scoring on result-set match, normalized into an order-independent, numerically tolerant multiset, means the brand claim holds literally: the kit catches wrong answers, it does not enforce one blessed query.
The review gate is a thin blocking wrapper, not a new scorer
The gate adds zero new comparison logic over the eval harness's scoring engine. It adds exactly two things: a blocking exit code and a human-readable reason. Golden-set mode runs the full suite for CI; candidate mode gates one agent-generated query against a hand-verified case.
A structural governance axis, separate from correctness
Result-set matching alone has a blind spot: a query that hardcodes the right literal, or reaches past the governed mart into a raw table, can still return the correct answer. The governance axis checks the derivation, not just the destination, so a memorized constant or a raw-layer leak fails even when the number is right.
Four governing modeling rules, machine-enforced
Staging is the anti-corruption layer, intermediate holds internal abstractions, marts are the public contract. These rules are written where the agent reads context and enforced by a layer-dependency lint at the gate, so a model that reaches past its layer is caught structurally, not just by an eyeball review.
Named failure-mode taxonomy
A binary pass or fail is enough to block, but not enough to diagnose. Every gate failure is attributed to a stage (parse, governance, execution, correctness) and a named mode from a single registry, so two candidates that both fail are visibly different problems: a hardcoded constant is not a raw-layer leak is not a wrong number.
Stack
Python, dbt Core, DuckDB, sqlglot, uv
Status
Active. Six sprints shipped, context pack, eval harness, review gate, governance axis, and failure-mode taxonomy all in place.
This repository is private. Email bharath@bharathk.dev for access.
Back to work