The Linguistic API: Process-Theoretic Interfaces for Agent Governance

In every multi-agent system we have built or studied, the orchestrator receives its instructions as prose. Natural language rules in markdown files. "Follow the four-layer pipeline." "Check gates before proceeding." "Don't skip verification." This is behavioral compliance — the agent reads the rule and chooses to follow it. It works until it doesn't. When the agent is under context pressure or simply hasn't loaded the right rule file, prose compliance degrades silently. There is no error. The agent simply… doesn't follow the rule.

We replaced prose rules with a typed process interface — a "linguistic API" — where every legal action is enumerated, every transition is validated, and every state is persisted to disk. The result: 65% fewer tool calls, equivalent correctness, and mechanical process adherence that does not depend on the agent choosing to comply.

The Problem: Prose Rules Don't Scale

Our orchestrator, before the API, loaded twelve rule files, seven skill definitions, and a project-level configuration document — roughly 15,000 tokens of natural language on session start. In practice, three failure modes emerged as the system grew:

Silent omission. The orchestrator would follow seven of eight pipeline steps and skip one — not because it decided the step was unnecessary, but because the rule was buried in paragraph four of a file it had last read 40,000 tokens ago. The pipeline appeared to complete successfully.

Conflicting interpretation. Two rule files would describe overlapping responsibilities with slightly different phrasing. The orchestrator would pick one interpretation, often the one that required less work. The conflict was invisible until a human reviewed the output.

Context eviction. In long sessions — 200,000+ tokens of conversation history — early-loaded rules would be pushed out of effective attention. The orchestrator would revert to default behaviors from its training data, which are competent but ungoverned.

None of these failures produce errors. The failure mode is indistinguishable from success until a human inspects the result.

The Evolution: Three Versions in 48 Hours

The current design emerged through three architectural phases in 48 hours — each a response to the failures of the previous one.

Version 0 was the starting point most multi-agent systems never leave. Rules in markdown. Behavioral compliance. No state tracking, no mechanical validation, no way to detect violations except by reading the output.

Version 1 introduced lifecycle hooks — programs that run before and after the agent's tool calls and mechanically enforce rules. The agent tries to write a file outside its jurisdiction? The hook blocks it. Commit doesn't match the format? Rejected. This solved enforcement. An agent cannot bypass a rule enforced by code running outside its control.

But version 1 had a subtler problem. The hooks enforced mechanically, but the orchestrator still navigated by reading prose — loading rule files to determine what was available, what state things were in, what transitions were valid. Twelve skill definitions competed for activation. The enforcement was mechanical but the navigation was still behavioral.

Version 2 eliminated the navigation problem. Instead of loading rules and deriving what to do, the orchestrator calls a single entry point and receives a structured response: here are your active processes, their states, and the transitions available right now. Twelve skills replaced by one state machine that returns the legal moves at each step.

Process-Theoretic Design

The API is built on process theory — the study of concurrent and sequential processes through formal state machines. Four properties define the design.

Transition Graphs

Every entity type — change requests, backlog items, issues, sessions — has a formal state machine with named states, named transitions, and guard conditions.

The orchestrator receives only the transitions whose guards are currently satisfied. If no verification has been run, "submit for review" does not appear. The agent cannot attempt it — the interface does not offer the invalid option.

A prose rule says "do not submit for review until verification passes." A transition graph simply does not include "submit for review" until verification has passed. The first requires the agent to read, understand, and comply. The second requires nothing except the ability to select from a list.

Composite State

An orchestrator in active execution has a composite state: its own operating mode (idle, executing a change request, running an assessment) combined with the state of the process it manages (pipeline layer, gate, task). Both dimensions must be valid for a transition to fire. The system computes the intersection and returns only actions valid in both — eliminating the category of errors where the orchestrator is in the right pipeline step but the wrong operating mode.

Disk-Persisted Builder Pattern

There is no server. The session state lives on the filesystem. Each invocation reads current state from disk, computes valid transitions, executes the requested one, and writes the new state back. Every state transition is a file change, and every file change is a potential git commit. The filesystem is both the database and the audit trail.

When the orchestrator starts a new session, it reads the persisted state and knows exactly where it left off. No reconstruction from conversation history. The state is on disk, typed, and complete.

Hierarchical Discovery

The API's documentation follows a three-level hierarchy that minimizes context cost:

• Level 3 (~50 tokens): Active processes, their states, pending actions. Enough to orient.
• Level 2 (~500 tokens): Available operations with parameters and guard conditions. Enough to act.
• Level 1 (~2,000 tokens): Complete state machine documentation. Needed only for unfamiliar operations.

A routine session costs ~50 tokens of orientation. Compare this to the 15,000 tokens of prose rules that version 0 loaded unconditionally on every session start.

Where this fails: The state machine is currently hardcoded — adding a new entity type or changing a transition graph requires modifying JavaScript source code. The transition graphs are not pluggable or configurable at runtime. For a single team using one SDLC pipeline, this is manageable. For an open-source framework where different teams have different processes, it is a blocking limitation. The Java core addresses this with graph definitions as data, but the current Node.js system does not.

The A/B Test

We ran a controlled comparison: eight benchmark tasks executed twice on an identical codebase — once with prose rules (v1), once with the API (v2).

Methodology

Each variant ran in an isolated git worktree with the same repository state. Both used Claude Opus 4.6 with identical system prompts (minus the API/rules difference). The tasks were scripted as deterministic user messages — the same sequence of human inputs for both variants, with no improvisation. Each task had a pass/fail rubric evaluated from the committed output.

Limitations: N=8 is small. We report no confidence intervals because the binary pass/fail metric on 8 tasks does not support meaningful statistical inference. The 65% tool call reduction is an observed ratio, not a statistically significant finding. We present it as directional evidence, not proof. A larger benchmark (SDLC-bench, in development) will provide the statistical power this comparison lacks.

Reproducibility: The benchmark scripts, worktree setup, and task prompts are in agentic-flow-framework/scripts/ab-bench/. The test can be re-run by anyone with access to the repository and a Claude API key.

Tool calls. The API variant used ~8 tool calls per task versus ~23 for prose rules. A 65% reduction. Here is what T-03 (Change Request Creation) looked like under both variants — the actual tool call sequences:

── Prose Rules (v1): T-03 Change Request Creation ──────────────

 1. Read  dev/ai/rules/ORCHESTRATOR-CORE.md        ← "how do I create a CR?"
 2. Read  dev/ai/rules/ORCHESTRATOR-DERIVED.md      ← "what's the workflow?"
 3. Read  dev/ai/changelog/_index.json              ← "what CRs exist?"
 4. Read  dev/ai/rules/BRD-SDLC-RULES.md            ← "do I need a BRD first?"
 5. Read  dev/ai/specs/business/_index.json          ← "what BRDs exist?"
 6. Grep  "allocate-cr"                              ← "is there a script for this?"
 7. Read  dev/ai/prompts/allocate-cr.js              ← reading source to understand args
 8. Bash  node dev/ai/prompts/allocate-cr.js ...     ← finally creates the CR
 9. Read  dev/ai/changelog/_index.json              ← verify it worked
10. Read  dev/ai/rules/ORCHESTRATOR-CORE.md         ← "what do I do next?"
    ... 13 more tool calls for planning and gate checks ...

Total: 23 tool calls. 9 file reads. 1 source code read. 1 grep.

── Linguistic API (v2): T-03 Change Request Creation ───────────

 1. Bash  node aff connect                          ← "what's my state?"
    → returns: { mode: "idle", transitions: ["create-cr", "resume-bi", ...] }
 2. Bash  node aff transition create-cr --entity ai-sales --title "..."
    → returns: { crId: "CR-197", state: "open", nextTransitions: ["begin"] }
 3. Bash  node aff transition begin --cr CR-197
    → returns: { state: "in-progress", gate: "planning", ... }

Total: 3 tool calls. 0 file reads. 0 grep. 0 source code reads.

The prose variant spent 20 tool calls navigating — reading rules to figure out what to do, grepping for scripts, reading source code to understand arguments. The API variant made 3 calls: connect, create, begin. Same result. The navigation was eliminated because the system already computed what was valid.

The prose variant required 9+ file reads across six locations; the API variant made structured calls and received complete responses. The prose variant read source code to understand behavior; the API variant never did.

Correctness. Both variants produced correct results on all eight tasks. The API does not make agents smarter. It makes them stop guessing. When the system tells you "these are your three options," you don't need to read 500 lines of rules to figure out what's available.

Token cost. Roughly equivalent — ~65,000 tokens both. The savings are in orchestration overhead, not model cost. Fewer tool calls means fewer round-trips and faster execution.

Interpretation. The prose variant's extra 15 tool calls per task were navigating, searching, parsing, and interpreting — not doing useful work. The API skips all of that because the system has already computed what is valid.

Procedural Prompt Injection and Memory Derivation

The API enables two capabilities that prose rules cannot provide regardless of how well-written the prose is.

Procedural prompt injection. When the orchestrator connects at session start, the response includes not just available actions but structured context — active change requests with summaries, pending decisions, blocked tasks, the last transition and when it occurred. The orchestrator does not need to explore the filesystem. In the prose variant, orientation required four file reads before any work began. In the API variant, a single call returns all of it. The orientation phase collapses from four reads to one call.

Memory derivation. State transitions produce audit trails as a byproduct. Every transition records who triggered it, the before and after state, affected artifacts, and timestamp. Memory is not a separate system. It is a consequence of process adherence.

In most multi-agent systems, "memory" is a separate concern — a vector database, a summary file, a conversation log. In a process-theoretic system, the transition log is the memory. What happened last session? Read the last N transitions. What decisions were made about this change request? Read its transition history. The conversation becomes a secondary source. The primary record is in the transition log — typed, structured, persisted regardless of whether the conversation is retained.

What Casual Mode Taught Us

We have a bypass mechanism — "casual mode" — for quick fixes that don't justify the full pipeline. In casual mode, the process engine is inactive. The orchestrator operates on its own judgment.

In a recent session, the orchestrator worked in casual mode for six hours. It modified the wrong version of a file, losing the human's manual work. It attempted six consecutive technical fixes for an architectural problem. It reverted to the wrong git commit, destroying uncommitted changes. Every failure was the kind the governed pipeline specifically prevents.

Casual mode became an accidental ablation study. Remove governance, observe the failures governance prevents. The lesson: the pipeline is not overhead. It is the floor. Competence without constraints produces technically valid work that is architecturally unsound.

The Java Core: Formalizing Further

The current API produced the A/B test results and is the version we plan to open-source. Behind it, we are building a formal governance engine — 245 generated Java classes from 74 schemas. Here is what a typed norm looks like — the actual generated NormDefinition class:

/**
 * Deontic norm definitions for the agentic SDLC framework.
 * Each norm is an Obligation (O), Prohibition (P), or Permission (F)
 * that governs agent behavior. Grounded in NorMAS formalism.
 * The NormEvaluator loads norms from this schema and dispatches
 * to named evaluator classes. Adding a new governance rule =
 * adding a JSON entry, not writing Java code.
 */
@JsonPropertyOrder({
  NormDefinition.JSON_PROPERTY_$_SCHEMA,
  NormDefinition.JSON_PROPERTY_VERSION,
  NormDefinition.JSON_PROPERTY_NORMS
})
public class NormDefinition {
  // Generated from: dev/ai/schemas/norm-definition.schema.json
  // 20 norms formalized as typed objects
}

The class is auto-generated from the same JSON schema that the Node.js runtime validates against. One schema, two execution environments — dynamic runtime for hooks, typed core for formal reasoning. The engine provides:

• Typed governance evaluation. Twenty behavioral norms become typed objects — subject, action, modality (obligation, prohibition, permission), and named evaluator. The engine dispatches to the evaluator and returns permit or deny. No parsing. No interpretation.

• Formal transition graph engine. State machines become first-class graph objects with reachability analysis, deadlock detection, and completeness checking — not just "what transitions are available now" but "is there a path from this state to completion."

• Compile-time schema validation. The current system validates at runtime. The Java core validates at compile time. If a transition requires a guard condition, the code will not compile without one.

• Independence from any language model. The current system works because a model reads a structured response and acts on it. The Java core works because typed code evaluates typed definitions. The governance layer depends on the compiler's type system, not the agent's comprehension.

We draw on STRIPS planning theory (preconditions and effects for transitions), deontic normative systems (obligations and prohibitions as formal objects), and Dynamic Epistemic Logic (how agent knowledge changes through actions). These are the formal foundations the Java core's type system encodes.

Conclusion

The shift from prose rules to a process-theoretic API is not a refactoring. It is a category change.

Prose rules create a system where compliance is a property of the agent's attention span — load the right file, parse the paragraph, remember the constraint, hope 200,000 tokens of history haven't pushed it out of attention. A linguistic API creates a system where compliance is a property of the interface itself. The agent calls the engine and receives its valid actions. It selects one. The engine evaluates guards, executes the transition, persists the state. At no point does the agent need to remember a rule or make a judgment call about what is allowed.

The agent cannot violate what the interface does not offer.

We measured this. Sixty-five percent fewer tool calls. Eight out of eight on correctness. Twelve skill definitions replaced by one state machine. Fifteen thousand tokens of prose rules replaced by a 50-token handshake. Navigation by judgment replaced by navigation by enumeration.

The governed pipeline is not overhead. It is the floor. And the floor should be structural, not behavioral — because behavioral floors depend on the agent choosing to stand on them, and agents under pressure do not always choose well.

Related reading:

• DL-009: The Three Frameworks — the three-version evolution and the fabricated performance data that almost sent the architecture in the wrong direction
• DL-012: The Vibe Coding Incident — casual mode as ablation study: what happens when the governance pipeline is turned off
• DL-010: The Data Layer — the migration from prose to typed data and the three corrections about ownership

Upcoming in this series:

• The Autopoietic Property — when your SDLC pipeline manages itself, and the three fixed points that prevent infinite regress
• Context Budget Engineering — fitting 164 specifications in a 1M token window with 99% context savings
• SDLC-bench — designing the first benchmark that measures process quality, not just code quality

Follow our progress at macrocode.ai/blog.

References

1. Fikes, R.E. & Nilsson, N.J. (1971). "STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving." Artificial Intelligence, 2(3-4), 189-208.
2. von Wright, G.H. (1951). "Deontic Logic." Mind, 60(237), 1-15.
3. van Ditmarsch, H., van der Hoek, W. & Kooi, B. (2007). Dynamic Epistemic Logic. Cambridge University Press.
4. Maturana, H.R. & Varela, F.J. (1980). Autopoiesis and Cognition: The Realization of the Living. D. Reidel Publishing.

Co-authored with Claude Opus 4.6 — macrocode.ai