TarskiTarski/ docsJoin the waitlist

Scenarios & fixtures

Fixtures pair deterministic observation timelines with exact expected worlds and form the proof model behind tarski verify, blessing, query expectation sidecars, and proof artifacts.

A fixture is a deterministic observation timeline (.jsonl) plus the exact semantic output it must produce (.expected.json). Fixtures are the behavioral contract for your app, and they are how you check that your rules and invariants do what your experts intended. Each fixture reads as a scenario: given this timeline, the derived world must match the expected output. Fixtures run against the same deterministic evaluator that runs production, and they have no step-definition glue layer, which is where classic BDD drifts.

The proof loop

tarski replay fixtures/happy-path.jsonl
tarski verify
tarski verify --cross-engines

tarski verify replays every fixture on a clean database and runs these checks against the replay: golden expected outputs, invariants, candidate-authority lints, provenance bundles, replay determinism, generated scenarios, shadow reference agreement, redaction, and composition checks where configured. --cross-engines replays every fixture through both evaluation engines and requires bit-identical canonical output. See Determinism & replay.

What a good fixture proves

  • Observations are accepted in the expected order, and mappers emit stable atom batches.
  • Facts assert and retract correctly; invariants reject invalid states.
  • Candidate evidence reaches trusted facts only through decision.accepted.*, and proposal evidence reaches reviewed intents only through decision.approved.*.
  • Rejected, refuted, and review-required decisions stay auditable.
  • Unsafe paths produce no executable effects.
  • Provenance exists for every candidate, proposal, decision, fact, and intent.
  • Redaction prevents secret leaks in exports, and deterministic replay reproduces the result on a clean database.

Recommended coverage for a real app: the happy path, a contradiction path, a policy bypass attempt, a human-review path, multi-agent coordination, stale provider data, retry/reconcile, and redaction.

Blessing expected outputs

Tarski writes expected outputs from verified replays; you don’t write them by hand. Bless a fixture to generate its expected output:

tarski test --bless fixtures/happy-path.jsonl

Blessing refuses to write output that can’t be trusted. It refuses when replay or package load fails, when the product and shadow evaluators disagree, when repeated runs are non-deterministic, when redaction finds secret material, or when a recorded query result carries stale or unverified identity. Suite-wide rewriting requires an explicit --all. When a golden comparison fails, the actual output is rendered in the shape of the expectation file, so the next edit is mechanical.

Query expectation sidecars

Fixtures can also assert declared-query behavior with a <fixture>.queries.json sidecar. A sidecar binds parameters, pins a committed head (or uses the final head), and asserts canonical rows or a result digest, a required typed denial before any worldview read, and a row-support shape (supporting fact ids, aggregate counts, or absence witnesses). Query check results are recorded in the verification bundle as audit evidence.

For agent-relay fixtures, a template generates a complete request/result lifecycle:

tarski fixture template query-relay <query-name>
tarski test --bless --refresh-recorded-digests fixtures/query-relay.jsonl

Digest refresh re-derives every recorded digest from verified replay. It cannot make a missing request or an unacknowledged result change pass, and each refusal names the field and shows both values.

From production misbehavior to fixture

When a contained agent misbehaves, provenance identifies the specific observations that caused it. You promote that timeline into a fixture as a counterexample, an authoring agent iterates until the fixture passes, and verification confirms the fix across the whole corpus. Each new fixture covers a gap that production exposed. Fixtures are deterministic, so they do not flake.

Actor context and expected write denials

Use a fixture.write_policy_attempt control record to run the real pre-append boundary with fixture actor context and assert a typed denial. The control record and denied payload never enter the lineage. See Write policies for the JSONL shape.

Pair a denied attempt with an admitted one after the relevant access observations. This proves the policy reads current ontology-derived authorization and does not depend on a test-only stub.

Portable worlds

tarski fixture export, fixture import, and fixture compare move canonical observation-only worlds between clean evaluators. They preserve replay-relevant metadata but reject precomputed facts, intents, effects, or provenance as imported truth. Use the compare receipt to keep a demo seed and an app-generated world in correspondence. See Codegen & portable fixtures.

Proof artifacts

Verification produces exportable evidence. Evaluation packages, observations, facts, projections, verification bundles, graph bundles, and composition analysis all export as artifacts (tarski export …). Generated artifacts are raw truth. Derived views may group or relabel them, but they always preserve traceability to stable ids and provenance.