Anatomy of a production agent workflow engine

The five step types

• llm_call — model, system, prompt template with {{var}} interpolation, output_var.
• tool_call — one of {web_search, http_request, json_transform, custom}. Returns a hashable object.
• branch — expression evaluated against state, picks one of N children.
• transform — deterministic string/object transform.
• human_gate — pauses until an out-of-band signal.

Everything is composable. The engine iterates steps, updates state, emits evidence.

State management

State is a flat dict. Every step reads and writes named variables. {{task_input}} is auto-populated at start. Interpolation is intentionally simple — no Jinja escaping surprises.

Evidence emission

Every step emits at least one evidence object:

• LLM_LOG — links to the gateway audit-chain entry.
• TOOL_OUTPUT — content hash + short preview.
• TRANSFORM — small enough to inline.

The evidence bundle at the end is what unlocks the accept button.

Cost tracking

Each llm_call returns a cost estimate from a maintained price table. Sum across steps → total job cost. Compare against manifest's spend_cap_usd and abort with a graceful error if exceeded.

Retries

Retry llm_call on transient errors (rate-limit, 5xx). Never retry tool_call unless the tool declares it safe (idempotent GETs, yes; POST with side effects, no). Never retry human_gate.

Concurrency inside a step

llm_call can fan out to N parallel provider calls (majority vote, first-to-answer). This is where quality gains hide.

The failure mode nobody catches

{{}} interpolation with an undefined variable. Silent empty string. Suddenly your prompt is "Analyze:" with no content and the model happily invents an answer. Fail loudly on missing variables — always.