TarskiTarski/ docsJoin the waitlist

Storage

Content-addressed blob storage with backend invariance, quotas, and GC, plus semantic tiering that keeps observation history at object-storage cost.

Tarski separates two storage concerns: blobs (large raw content referenced by observations) and observation history (the append-only log itself, kept forever and tiered by semantics).

Blob storage

Large raw content, such as uploads, fetched documents, and archived reports, is stored as content-addressed blobs. A blob’s identity is sha256:<hex> of its canonical bytes, computed before any backend write. Observations and receipts carry digest-only BlobRefs.

Where bytes physically live is deployment configuration with no semantic authority. Replaying the same timeline must produce byte-identical digests, receipts, derived facts, and exports on every supported backend. The supported backends are sqlite (zero-config local default), memory, mock (deterministic test double with scripted faults), and s3 (any S3-compatible endpoint).

[storage]
blob_backend = "s3"
max_blob_bytes = 26214400

[storage.s3]
bucket = "my-app-blobs"
endpoint = "https://s3.example.com"
access_key_env = "MY_S3_ACCESS_KEY"    # credentials by env-var name only
secret_key_env = "MY_S3_SECRET_KEY"    # inline values are rejected at load

Selection priority is flag > env (TARSKI_BLOB_STORE) > file > sqlite, with a --blob-store flag on dev, serve, replay, and verify. A configured S3 target runs a write/read/head/delete probe at startup and fails closed. It never falls back silently to another store. Managed Cloud ignores app-level storage configuration entirely, and the platform owns the target there.

Apps reach storage through declared blob.put / blob.get capabilities. Typed errors cover each failure: blob.size_limit_exceeded, blob.quota_exceeded, blob.not_found, blob.corrupt_content, blob.tombstoned, blob.restricted, blob.range_unsatisfiable.

Operate it with:

tarski stats                      # usage, egress, quota pressure
tarski gc --blobs --dry-run       # unreferenced-blob collection, tombstoned
tarski storage migrate --to s3    # copy with digest verification
tarski blob put ./report.pdf      # plus: hash, get, inspect

Client blob contract

@tarski/client 0.8.0 exposes generated upload, inventory, and metadata operations. client.downloadBlob() handles full or ranged raw-byte downloads. Each request is lineage-scoped and uses the same in-memory runtime or verified end-user bearer as other SDK calls.

Possessing a digest does not authorize access to its blob. The runtime binds the authenticated organization, tenant, workspace/lineage, active redaction policy, and caller scope before touching storage. A digest from another tenant cannot be used to probe existence. Listing is filtered before pagination, and restricted content may leave its digest visible while withholding bytes.

The default per-blob cap is 256 MiB; deployments may set a smaller cap. The workspace hard quota is dedup-aware: uploading bytes already stored under the same digest is idempotent and does not consume quota again. Hosted transfer and egress quotas can also return typed, retryable throttling with a retry window.

Client-visible failures are stable:

Code HTTP Meaning
blob.size_limit_exceeded 413 This object exceeds the configured per-blob cap
blob.quota_exceeded 413 The workspace hard quota would be exceeded
blob.not_found 404 No authorized object resolves for the digest
blob.restricted 403 Redaction/authorization permits metadata but withholds bytes
blob.tombstoned 410 Physical purge is recorded by a durable tombstone
blob.range_unsatisfiable 416 The requested range lies outside the byte length
blob.corrupt_content 502 Stored bytes failed digest verification and were not served

Use capability discovery before showing an upload control. An authority that omits the generated blob operations resolves to unsupported_runtime, and the SDK does not guess an older route.

Quota pressure reports as under, soft_exceeded, or hard_exceeded, with dedup-aware usage. Studio’s Storage surface adds a blob browser, uploads, downloads, and dry-run-gated reclamation.

Semantic tiering

The observation log is canonical truth, so Tarski keeps all of it, at object-storage cost. Semantic tiering makes that economical without a recency heuristic. Sealed, hash-chained archive segments carry semantic coverage, computed from the same canonical mappers that replay uses. Coverage records which namespaces, entity keys, and strata each segment can affect, under which evaluator digest.

Cold reads request history in ontology terms and fetch only matching segments. A read can ask for everything (replay), a namespace reduct (a declared query’s read set), or an entity neighborhood (provenance traversal). The rules are conservative:

  • Pruning fails safe. Absent coverage, a foreign evaluator digest, or an overflowing neighborhood index always degrades to fetch-everything. Pruning is only an optimization, and correctness does not depend on it.
  • Integrity fails closed. Digests verify before decoding, and tampered bytes fail.
  • Identity is tier-invariant. Archive replay produces the same revision digest as hot replay, and pruned query execution produces the same result digest as full hydration. Every hydration returns a receipt (segments fetched, observations, bytes).

tarski stats folds validated segment manifests into per-lineage archive cost lines: segment count, archived observations, archived bytes, and floor bytes-per-observation.