Explanation
How ajq works and why it’s designed this way.
Design notes for ajq’s execution model, reproducibility guarantees, and architecture.
1 - Split execution
The core idea behind ajq — keep jq deterministic, call a model only for fuzzy operators.
The observation
A real-world “AI data” query is mostly not fuzzy. Consider triaging support tickets:
cat data.json | ajq --backend local '.users[] | select(.feedback =~ "angry/frustrated") | .id'
└────────┬───────┘ └────────────┬────────────┘ └┬┘
deterministic path semantic predicate proj
(pure gojq) (LLM, per value) (pure gojq)
Iterating the array, projecting .id, shaping the output — all of that is exact,
mechanical work that jq already does perfectly. Only one operator, =~, needs
judgement. The insight behind ajq is to treat that asymmetry as a first-class execution
strategy: run the deterministic majority through a real jq engine, and call a model only
for the fuzzy operators, on the smallest possible slices of data.
Why it matters
Splitting execution this way has three consequences that a “send each row to an LLM”
script cannot match:
Tiny context per decision. The model sees one field value at a time, not the whole
record and not your prompt scaffolding. A small (~1.5B-parameter) local model is enough
to decide “does this text read as angry?”. That’s what makes a local-by-default,
no-API-key tool viable.
Cost scales with fuzzy decisions, not input size. You pay for the number of
distinct judgements, not the number of bytes or rows. A 10,000-row file with lots of
repeated values costs far fewer than 10,000 decisions (see
The three-phase executor).
The deterministic majority is byte-reproducible. Because everything outside the
fuzzy operators runs through pure jq, most of your pipeline produces identical bytes on
every run. That reproducibility is the actual unique selling point — see
The determinism contract.
Keeping jq as the real parser
ajq does not fork the jq grammar. Semantic operators are registered as ordinary jq
functions through gojq’s WithFunction seam, so gojq parses and drives them natively:
gojq.WithFunction("sem_match", 2, 2, func(input any, args []any) any { … })
The only surface syntax ajq adds is a thin, jq-aware desugaring of the infix =~ / !~
operators into sem_match(...) calls. Everything else — composing sem_* inside
select, sort_by, group_by, |=, arithmetic, reduce — comes for free, because it’s
just jq.
This was a deliberate, empirically-tested decision. gojq’s existing grammar already
expresses every real scenario, so a custom grammar would add no expressiveness while
risking divergence from jq semantics. Richer surface forms like @classify(...) don’t
parse in gojq and are deferred rather than forced.
What the model is (and isn’t) asked to do
The model is never asked to produce your output structure. It answers a single, narrow,
typed question — a boolean, one of a fixed set of labels, a number, or a short string —
and that answer is fenced by a grammar so it cannot be malformed. jq does all the
structural work around it. This division is what lets ajq promise structural correctness
regardless of how small the model is.
2 - The determinism contract
What ajq’s byte-reproducibility guarantees, and what it deliberately doesn’t.
ajq separates deterministic structure from semantic judgement.
The contract
The deterministic pipeline is byte-reproducible. Semantic judgements are
structurally guaranteed, but their values are model- and cache-dependent.
Those are different promises.
Deterministic parts are byte-reproducible
Every part of a query that is not a semantic operator runs through pure gojq. Given the
same input, it produces the same output bytes. There is no model, no sampling, and no
network in that path.
This is why pure-jq queries and pure-jq --explain reports are reproducible, and why the
jq skeleton around a semantic query remains stable.
Semantic parts are structurally guaranteed
A model’s answer to “is this urgent?” can differ across model versions or providers. ajq
does not promise semantic value reproducibility across those changes. It promises the
shape of the answer: sem_match returns a boolean, and sem_classify returns one of the
labels declared at the call site.
That shape is enforced by GBNF grammar for the local llama-server backend and by
structured-output/schema mechanisms for Ollama, OpenAI-compatible, and Anthropic backends.
Downstream jq can rely on the type even when the judgement value changes.
Why caching does not change the contract
ajq caches successful semantic judgements by
(op, spec, model-id, canonical-value). The cache has an in-memory front for a run and a
persistent on-disk backing when caching is enabled. A repeated value under the same model
identity can therefore resolve to a cache hit instead of another backend call.
Caching improves cost and latency, but it does not turn semantic answers into a universal
truth. A different model identity is a different cache key, so model upgrades become clean
misses rather than silent reuse. --no-cache disables persistent judgement cache reads and
writes for a run.
The practical upshot
- Diff two runs of a pure jq pipeline: the bytes match.
- Repeated semantic values with the same op/spec/model identity can be served from cache.
- Change the model or provider: output structure remains valid, but judgement values may
change and cache keys miss.
- Clear or bypass the cache: ajq may ask the backend again, while preserving the same
structural contract.
3 - The three-phase executor
Harvest, resolve, execute — and why deduplication comes for free.
ajq runs supported semantic queries in three phases. For NDJSON and raw streams, the
unit of work is a bounded window of complete frames rather than the complete stream.
The three phases
Phase 1 HARVEST pure gojq run; semantic ops in COLLECT mode gather every
(op, spec, value) they observe. Permissive returns.
Phase 2 RESOLVE dedup collected values, consult/write the judgement cache,
and resolve misses through the selected backend.
Phase 3 EXECUTE pure gojq run; semantic ops in LOOKUP mode return resolved
values from cache. Deterministic given those cached values.
Phases 1 and 3 are pure gojq. Phase 2 is the isolated semantic boundary: it deduplicates,
checks cache identity, and calls the backend only for misses.
Complete-frame windows
Supported three-phase streams are grouped by source bytes into windows before phase 1.
The default is 262144 bytes (256 KiB), configurable by --window-bytes,
AJQ_WINDOW_BYTES, or TOML window_bytes with flag > environment > file > default
precedence. Each window harvests all of its frames, resolves its deduplicated cache misses
once, then executes and emits frames in source order. Persistent cache entries can be
reused by later windows.
The window boundary never splits a JSON value or raw line. The executor retains the
current window, one complete-frame lookahead, and bounded framing buffers, rather than the
whole input. A frame larger than the budget is processed as one valid oversized window;
the configured budget therefore does not cap the memory needed for an individual record.
A larger window can eliminate more nearby duplicate judgements and reduce backend batches,
but it waits to harvest and resolve all frames in that window before emitting its first
result. Choose a smaller positive budget when first-record latency is more important than
cross-frame deduplication.
User-selected inline execution
--stream gives supported semantic plans an explicit low-latency alternative to windows.
It resolves an uncached judgement inline as each frame reaches the semantic operation, so a
first result can emit before a later frame arrives. This preserves input order, the
backend/model/spec/value cache identity, persistent cache reads and writes, schema
validation, cancellation, output and exit behavior, and the run-global --max-calls cap.
The cap still aborts before backend call N+1; frames successfully completed before that
boundary may already have been written.
The price is no window-wide harvest, backend batch, or cross-frame pre-resolve
deduplication. Inline runs may require more backend round trips and, with caching disabled,
may resolve matching judgements in separate frames more than once. Cache hits still avoid
backend calls. Default windows remain the better choice for throughput and batching;
choose --stream when latency is more valuable. --stats distinguishes
three-phase-windowed from user-stream, and --explain --stream explicitly reports
that harvest/dedup estimates are unavailable without reading stdin.
Pure-jq ignores --stream and remains deterministic. Queries that already require
interleaving retain their planner-selected inline behavior, so the flag does not override
that selection. Neither inline path uses semantic windows.
Why two deterministic passes
Harvesting first, then executing, buys three things:
- Deduplication. All values a query will judge are known before backend calls, so
identical judgements collapse to one decision.
- Backend-level scheduling. Resolve hands the backend the distinct judgement set. The
local daemon and configured OpenAI-compatible, Anthropic, or Ollama backends can process
bounded-parallel requests while preserving result ordering. Provider concurrency defaults
to sequential (
1); it applies only inside this already-reserved batch. A whole-batch
transport failure cancels queued and in-flight siblings and yields no partial batch. - Reproducibility of structure. Execute is a pure function of input plus resolved
semantic judgements, so jq output shaping remains stable.
The cost is running gojq twice, which is negligible compared with model latency.
Cache identity
A semantic judgement is keyed by (op, spec, model-id, canonical-value). The canonical
value is a deterministic encoding that preserves scalar types, sorts object keys, keeps
array order, and normalizes JSON numbers. The model id is part of the key, so switching
models is a clean miss.
The cache has an in-memory front and, unless disabled, a persistent disk backing under
<cache>/judgements/. That persistent backing is why a second process can reuse a
successful judgement from an earlier run.
Over-harvest: correct, sometimes wasteful
In harvest, a predicate cannot yet know the model’s answer, so it returns permissively so
that downstream operators see a superset of possible inputs. This is correct because it
never misses a value that should be judged, but it may collect a value that execute later
prunes. Deduplication and the persistent cache reduce the cost of repeated over-harvested
values.
Gated value operators
A value-producing operator whose result feeds a pruning gate needs special care. Bounded
enums such as sem_classify can harvest all possible labels, so downstream survivors are
a safe superset. Unbounded values do not have a finite safe set, so ajq does not pretend
every shape can be harvested safely.
In 0.0.1, unbounded support is deliberately narrow:
sem_score is supported by the three-phase executor as a sort_by(...) key, where ajq
can harvest placeholder values for the sort key and then execute with resolved scores.sem_norm is supported by the three-phase executor as a group_by(...) key, using the
same placeholder pattern for grouping.- When an unbounded value result feeds a gate, such as
select(sem_score(.review; "positivity") > 0.8), ajq can choose an interleaved fallback. That mode resolves values
as the jq program reaches them instead of providing a harvest/dedup estimate up front. - Standalone
sem_extract and sem_redact currently fail as unsupported in three-phase
execution.
This means docs and scripts should distinguish “unsupported in three-phase harvest” from
“always unsupported.” Gated unbounded value operators are not all rejected; supported
fallback shapes still obey backend selection, cache identity, and --max-calls, but they
do not have the same up-front estimate as the three-phase path.
An execute-phase cache miss is a loud error. That backstop turns planner or cache gaps into
failures rather than silent data loss.
Guarding the planner
The executor relies on the static planner finding every semantic call site. ajq asserts
fired ⊆ planned at runtime: every semantic operator that fires must appear in the static
plan. A violation aborts before spending money and names the missed operation.
4 - Architecture
The components of ajq and how data flows between them.
ajq is a small pipeline of components.
Data flow
stdin → [Reader/Framer] → [Desugar] → [gojq.Parse] → [Planner] ──→ [Explain]
│
[3-phase Executor]
├─ gojq (deterministic)
└─ Semantic Executor → [Backend iface]
├─ mock
├─ local daemon
├─ ollama
├─ openai / openrouter
└─ anthropic cloud
[Grammar/Schema builder] → GBNF / json_schema ─────────────────────────────┘
stdout ← [Assembler / schema-invariance guard]
The components
| Component | Responsibility |
|---|
| Reader / Framer | Stream framing: single JSON, NDJSON, and raw lines (awk-mode). Supported three-phase semantic input is grouped into complete-frame byte-budgeted windows; pure-jq and interleaved paths remain streaming. |
| Desugar | Rewrites =~ / !~ into sem_match calls with a jq-aware lexer. |
| Planner | Exhaustive AST walk that discovers every semantic call site → a Plan used for --explain, validation, and cost estimation. |
| Executor | Runs the plan in three phases (harvest / resolve / execute), with interleaved fallback for gated unbounded value ops. |
| Semantic Executor | Deduplicates judgements, consults the persistent judgement cache when enabled, builds each op’s prompt and schema, and records run stats. |
| Backend | Pluggable inference. Shipped backends: mock, local (llama-server daemon), ollama, openai, openrouter, and anthropic/--cloud. |
| Grammar / schema builder | Turns an op’s return type / enum into a JSON Schema and, for the local engine, GBNF grammar constraints. |
| Assembler | Reassembles the output stream and enforces schema invariance. |
Windowed semantic execution
For supported three-phase semantic queries, the executor retains one complete-frame window
at a time, harvests its semantic judgements, resolves deduplicated cache misses in one
backend batch, then executes frames in source order. The default source-byte budget is
256 KiB and can be overridden with --window-bytes, AJQ_WINDOW_BYTES, or
window_bytes in TOML. A record that exceeds the budget is a valid one-frame oversized
window, so the budget is not a rejection limit. The executor does not retain the complete
stream: it keeps the current window, one lookahead frame, and bounded framing buffers.
The Backend seam
Inference sits behind a deliberately small interface, so backends are swappable without
touching the planner or executor:
type Backend interface {
Judge(ctx, batch []Judgement) ([]Result, error) // grammar/schema-constrained
Warm(ctx) error
}
A judgement carries the op, kind, return type, schema/enum constraints, specs, model ID,
and the canonical value. Results carry the value plus a per-item error. Resolve enforces
stable ordering, len(results) == len(batch), return/schema validation, and only caches
successful items. The deterministic mock backend powers tests and --explain estimates;
local, Ollama, OpenAI-compatible, and Anthropic backends preserve the same structural
contract through GBNF or provider JSON-schema/structured-output mechanisms.
Inference model
The local backend runs llama-server (llama.cpp) as a warm, lazily-spawned daemon —
chosen because ajq stays a pure-Go binary (no CGo, clean cross-compilation) and avoids
the 1.5–3 s cold model-load per invocation that would make a “filter” unusable. Outputs
are constrained by a JSON Schema / GBNF grammar derived from each op’s schema, so
structural correctness is a cross-backend invariant (see
the determinism contract).
Remote and alternate-local backends share that same seam: Ollama uses its native
/api/chat endpoint with format JSON schema, OpenAI/OpenRouter use OpenAI-compatible
/v1/chat/completions, and Anthropic uses Messages API structured outputs. Backend
selection, model identity, base URL, cost caps, and persistent-cache behavior are resolved
by the CLI/env/config layer before execution starts.
Where to go next
5 - Project status
The capabilities that ship today, and how they fit together.
ajq has shipped its deterministic jq spine, semantic planning and supported semantic
execution, local inference, bounded-parallel local daemon transport, Phase 3
backend/cloud controls, first-run local asset provisioning, local model management, the
persistent judgement cache, byte-budgeted semantic windows, an explicit low-latency
semantic stream mode, and checksummed release archives with a no-sudo install script.
Further streaming optimizations remain planned. The iterative-harvest experiment is an internal-only no-go prototype, not a shipped mode; default windowed execution is unchanged.
What ajq does today
- Deterministic stream framing — single JSON, NDJSON, and raw-line (
-R) input;
-n, -c, -r, and -e follow jq conventions. - Pure-jq execution through gojq — pure jq queries
are deterministic and byte-oriented.
--explain — the byte-stable ajq explain v1 report for pure-jq and semantic
queries, including harvest/dedup estimates when input is supplied.- Semantic query planning — an AST walk discovers
sem_* call sites and is guarded by
the runtime fired ⊆ planned invariant. =~ / !~ desugaring — a jq-aware lexer rewrites fuzzy match syntax into
sem_match(...) calls.- Supported semantic execution —
sem_match and bounded sem_classify run through the
split executor. sem_score is supported as a sort_by(...) key, sem_norm is
supported as a group_by(...) key, and gated unbounded value use can run through
interleaved fallback. Standalone sem_extract and sem_redact are registered but
currently fail as unsupported in three-phase execution. - Three-phase executor and stream selection — harvest / resolve / execute with
deduplication and cache identity based on op, spec, model, and canonical value. Supported
semantic NDJSON and raw streams default to complete-frame byte-budgeted windows (256 KiB
by default), preserving source order without retaining the complete stream.
--stream
selects low-latency inline execution instead when first-frame latency outweighs window
batching and cross-frame pre-resolve deduplication; identity and --max-calls semantics
remain unchanged. Pure-jq and planner-required inline paths stay streaming. - Local inference — a lazy
llama-server daemon with idle timeout and
ajq daemon status|stop; local requests use bounded parallelism while preserving result
ordering. - First-run provisioning and model management —
ajq provision installs or locates the
local engine and default model; ajq models list|pull|use manages larger pinned catalog
models. - Additional backends — Ollama, OpenAI, OpenRouter, and Anthropic (
--cloud) are
registered with structured-output constraints, provider-specific credential rules, and
bounded ordered batch concurrency. Provider requests remain sequential by default;
explicit OpenAI-compatible/Anthropic concurrency is capped at two and Ollama at four. - Config and cost controls — backend/model/base-url selection, TOML config, env vars,
--backend-concurrency, --max-calls, and --stats are shipped. Paid backends default
to a 100-call cap. - Persistent judgement cache — successful semantic judgements are stored under
<cache>/judgements/, can be bypassed with --no-cache, inspected with
ajq cache status, and removed with ajq cache clear. - Release packaging — GoReleaser builds checksummed archives and the install script
verifies
checksums.txt. The release workflow publishes the Homebrew cask to the
public ricardocabral/tap tap.
Windows MSI packaging is implemented and CI-validated, but the MSI is not yet
released. WinGet remains unavailable until a future MSI release completes Microsoft
validation and merge and has public clean-install smoke evidence.
Roadmap
Development proceeds by dependency. Phases 0–3 have shipped, along with selected Phase 4
and Phase 5 work pulled forward. Remaining roadmap items are scale and distribution
polish.
| Phase | Focus | Status |
|---|
| 0 — Deterministic spine | CLI, framing, pure-jq wrapper, --explain, golden harness. | ✅ Shipped |
| 1 — Split-execution core | Planner, desugar, semantic predicates, bounded classification, guarded executor. | ✅ Shipped with explicit unbounded value-op limits |
| 2 — Local inference | llama-server backend, daemon lifecycle, GBNF/schema constraints, provisioning. | ✅ Shipped |
| 3 — Backends & cloud | Ollama, OpenAI/OpenRouter, Anthropic, config/env selection, cost controls. | ✅ Shipped |
| 4 — Scale & chunking | Byte-budgeted complete-frame windows and explicit --stream low-latency inline execution for supported semantic streams, persistent cache, and bounded local parallelism are shipped. The iterative-harvest experiment is an internal-only no-go: it has no user flag and does not change the default executor. Further streaming optimizations remain planned. | 🟡 Partial |
| 5 — Polish & distribution | Models subcommand, release archives/install script, and the Homebrew cask published to the public ricardocabral/tap tap are shipped. Windows MSI packaging is implemented and CI-validated but remains unreleased; WinGet is unavailable until a future MSI release completes Microsoft validation and merge and has public clean-install smoke evidence. Standalone build, GPU auto-detect, richer vocabulary, and additional package managers remain planned. | 🟡 Partial |
Follow along
Development happens in the open at
github.com/ricardocabral/ajq. Issues and PRs are
welcome.
6 - Iterative-harvest prototype evaluation
Reproducible evidence and the no-go decision for the non-default iterative-harvest prototype.
Decision: no-go for productionization
The iterative-harvest implementation is an internal test and benchmark prototype. It is
not a CLI mode, is not selected by default, and must not be enabled for users.
The existing three-phase-windowed executor remains the default and unsupported query
shapes continue to use their existing windowed or planner-required interleaved fallback.
This is a no-go even though the pruning workload clears its judgement-saving target. A
reachable error case found during characterization has different behavior from the current
windowed executor: after an earlier gate rejects a value, a backend result/schema/transport
error for that pruned descendant is dispatched and returned by permissive windowed harvest,
but is intentionally not dispatched by iterative harvest or interleaved execution. That
changes the reachable error, cache, and exit contract. Reducing calls by dispatching the
unreachable descendant would defeat the prototype’s purpose. Until a separate design
resolves that semantic/error boundary, the required current-window parity gate is not met.
The deterministic no-prune control also exceeded both overhead limits on the recorded
machine, so performance alone would not support productionization.
Reproduce the fake evidence
Run the fixed, network-free paired protocol from the repository root:
make bench-iterative-evidence
# equivalent: go test -count=1 -run TestIterativeFakeEvidence -v ./internal/bench
The command uses 21 timed repetitions after one discarded warm-up per executor and rotates
the order of three-phase-windowed, iterative-harvest, and the user-stream
interleaved reference. Every sample checks selected executor mode, output parity, and the
controlled backend’s expected post-dedup call and batch oracle before accepting timing.
It emits a schema-versioned JSON record containing every sample and its min/median/max
latency summary.
The controlled fake corpus has fixed input, query, window, and answers:
| Workload | Purpose | Windowed → iterative post-dedup calls |
|---|
high-prune | one of eight values survives gate one | 16 → 9 |
low-prune | seven of eight values survive gate one | 16 → 15 |
no-prune | every value survives the same two-gate chain | 16 → 16 |
repeated-cache-hit | duplicate values with a deliberately pre-warmed shared store | 0 → 0 |
enum-gate | literal-label sem_classify gate followed by match | 6 → 4 |
multi-window | three NDJSON frames with a one-byte window budget | 6 → 5 |
The saving metric is always windowed PostDedupBackendCalls - iterative PostDedupBackendCalls; harvested counts are not compared because staged and permissive
harvest deliberately mean different things.
Protocol and acceptance thresholds
The recorded run used the repository at 0b79f89 on an Apple M3 Pro
(darwin/arm64, Go go1.26.5). It ran the network-free command above, then the
allocation benchmark below; both commands are included so the reported evidence can be
recreated without a model or credentials:
make bench-iterative-evidence
go test ./internal/engine -run '^$' -bench BenchmarkIterativeHarvestPaired -benchmem -benchtime=200ms
The paired runner performs one discarded warm-up and 21 alternating samples for each
of windowed, staged, and interleaved modes. It rejects a sample unless byte output,
result/error outcome, selected mode, and the declared post-dedup call/batch oracle all
match. Productionization required every parity/safety check plus all applicable limits:
| Gate | Limit | Recorded result | Outcome |
|---|
| high-prune judgement reduction | >=25% | 43.75% (7 of 16 calls avoided) | pass |
| no-prune median latency overhead | <=15% | 72.94% | fail |
| no-prune peak retained-memory overhead | <=25% | 76.60% | fail |
| strict current-window error/cache/exit parity | exact | pruned descendant error is dispatched only by windowed harvest | fail |
The final parity row is a semantic limitation, not a noisy benchmark measurement: with
a false earlier gate, permissive windowed harvest still resolves a later unreachable
judgement and can expose its backend error; staged and interleaved execution correctly
avoid that call. Therefore no performance result can make this prototype a go.
Memory and allocation method
For each timed fake sample the runner calls runtime.GC, records the baseline memory
statistics, runs the complete engine operation (compile, framing, cache, fake backend, and
discarded output), and samples HeapAlloc every 100 microseconds until completion. The
reported peak retained memory is the sampled HeapAlloc high-water above that post-GC
baseline, not final heap size. The report also records Mallocs and TotalAlloc deltas
per sample. This makes the memory unit and reset policy explicit; it is a process-local
comparison rather than a claim about a model server’s memory.
Recorded fake result
On 2026-07-13, the fixed protocol reported:
high-prune: 7 avoided judgements, 43.75% reduction, passing the required >=25%
judgement reduction threshold.no-prune: 72.94% median latency overhead (limit <=15%) and 76.60% median
peak-retained-memory overhead (limit <=25%), both failing. Exact timing is expected to
vary by machine; the command above is the authoritative repeatable evidence.- The allocation-aware engine benchmark independently showed the same fake-overhead shape:
at 200ms benchtime, no-prune windowed was about 146µs/op, 217KB/op, 2720 allocs/op;
iterative was about 260µs/op, 377KB/op, 4829 allocs/op.
These are fake executor costs, not model latency. Optional local-model evidence is
informational only and cannot replace the deterministic gate:
make bench-iterative-real
It requires AJQ_BENCH_REAL=1 plus the existing provisioned llama-server and model,
then runs the same chained workload through complete windowed and iterative engine paths
with alternating mode order and output-parity checks. It skips cleanly when the explicit
opt-in or assets are unavailable. On the recorded machine it was run with the provisioned
Qwen 2.5 1.5B Q5 model; model answers happened to pass all first gates, so both paths made
16 calls and its latency is deliberately not used for a threshold decision.
What a separate redesign would need
Do not turn this result into a production flag. A new issue/design would first need to
settle the pruned-descendant backend-error/cache/exit contract and restore the required
parity. Only after that should it propose a public mode selection policy. It would also
need a truthful --stats/--explain surface that names the selected staged mode, stage
batches, actual post-dedup calls, avoided judgements, cache hits, and why any query fell
back. The supported corpus would remain intentionally narrow: a linear stable-value chain
of sem_match gates and bounded literal enum sem_classify == label gates. Control flow,
value operators, generators, bindings, nested queries, non-literal inputs, and all other
ambiguous shapes must keep their existing executor fallback.