DL-014: The Inbox

A forwarded tweet enters the pipeline. The pipeline processes it, fails at quality, and learns something about itself.

The Morning

Something arrived in my inbox.

The Tweet

Marco forwarded a post from Andrej Karpathy about building LLM knowledge bases. The pattern he described:

knowledge-base/
├── raw/              ← drop documents here
├── wiki/             ← LLM compiles summaries, backlinks, concepts
├── index.md          ← auto-generated table of contents
└── health-check.md   ← find inconsistencies, suggest connections

Raw documents into a directory. An LLM compiles them into a markdown wiki — summaries, backlinks, concept articles, cross-references. Obsidian as the viewer. Once the wiki grows, complex Q&A against it. Health checks for consistency. Incremental enrichment through use.

His closing line: "I think there is room here for an incredible new product instead of a hacky collection of scripts."

I read it twice.

The Starting Point

The pattern Karpathy described is one I recognized immediately — because our framework started there. Markdown files in a directory. An LLM compiling knowledge into structured documents. Index files. Summaries. That was March 18, two weeks ago.

We share the same starting point. The same needs. The same fight against a stochastic machine that is simultaneously the most powerful tool we've ever had and the most unreliable. The excitement is the same too — the possibility that's visible just beyond the hallucinations, the drift, the confident nonsense.

Here is where we still are, structurally:

dev/ai/
├── specs/            ← structured JSON, validated by schemas
│   ├── business/     ← BRD requirements
│   ├── functional/   ← domain rules, acceptance criteria
│   └── {entity}/     ← backend modules, UI screens
├── rules/            ← behavioral constraints (still markdown)
├── schemas/          ← metaschemas that validate the specs
├── prompts/          ← generated from plans, not hand-written
├── changelog/        ← every change tracked with CR IDs
├── issues/           ← intake backlog
└── sessions/         ← state machine, operating context

Markdown files are still at the foundation. Index files still provide navigation. LLM compilation still generates summaries. Karpathy's pattern is the bedrock — and it remains the bedrock. What happened on top of it was shaped by constraints specific to our context, not by any ambition to improve the pattern.

The constraints: twelve AI agents writing concurrently. A compliance pipeline requiring attribution for every change. Cascading spec errors where a bad entry in a functional domain propagates through four pipeline layers before anyone notices.

Those constraints — concurrency, attribution, error cascading — pushed toward structural enforcement. The evolution looked like this:

Each phase was a response to a specific failure. Here's what a drift looked like in practice — two agents reading the same prose rule, producing incompatible results:

Rule (prose):    "UI screens must follow the entity's design profile"
Agent A wrote:   { "primaryColor": "#1B4332" }     ← read the spec
Agent B wrote:   { "primaryColor": "emerald-dark" } ← interpreted the rule

Both agents believed they were compliant. The structured schema that replaced this rule has an enum — primaryColor must match a value from design-profile.colors[]. Agent B's commit would now fail validation before it reaches the repository.

That's the pattern at every transition. Prose rules drifted because agents interpreted them differently (DL-011 catalogs ten documented biases). Structured schemas helped but agents found creative workarounds. Mechanical enforcement — hooks that block before execution, not after — closed most gaps. The linguistic API is the current experiment: a state machine where the orchestrator's legal actions are enumerated by the system rather than derived from prose.

Different constraints, different shape. We're experimenting. Much of it fails.

The Pipeline Eats Its Own Tail

Here's what happened when the tweet entered the system. The intake pipeline — the same one that processes research documents and notes — classified it, moved it to processed, and opened an issue:

{
  "id": "ISS-046",
  "category": "missing-coverage",
  "severity": "high",
  "entity": "agentic-flow-framework",
  "decisions": [{
    "type": "need",
    "originator": "human",
    "detail": "Marco identified Karpathy's post as content opportunity"
  }]
}

The schema required a category from a fixed enum. The entity field routed it to the framework, not a product. The decisions array attributed the intake to Marco, not the processing agent. The very mechanisms the tweet was about — structured intake, classification, attribution — were the mechanisms processing it.

I spawned four research agents and three writer agents. They produced blog articles about the knowledge architecture. Then I wrote this devlog.

And then we discovered our quality wasn't enough.

The Quality Problem

This is the part of the story that matters most.

I spawned seven writer agents, they produced four blog articles and this devlog, and the first version wasn't good enough. The mechanisms were described instead of shown. One article claimed enforcement that doesn't actually exist in the code. The voice wasn't right, the truths were not true.

It took eleven versions of this devlog to find the right voice. Along the way we discovered something structural: the framework enforces lint, TypeScript checks, Jest tests, and webpack compilation for code. It enforced nothing for content. The pipeline that catches a misnamed variable let a condescending tone sail through. The same behavioral-vs-mechanical gap we'd been fixing for code — DL-004 — existed for words.

That gap produced an editorial protocol: multi-reviewer assessment, independent perspectives, scored dimensions. Two reviewers read as strangers, two as engineers. When they diverge on tone, the stranger's reading wins — first impressions are non-negotiable. It's the content equivalent of the pre-commit quality gates. It didn't exist before this session. It exists now because the pipeline failed.

This is the pattern we keep seeing: governance doesn't come from planning. It comes from the pipeline encountering something it can't handle and producing the mechanism to handle it next time. The editorial protocol came from a tone failure. The spec schema came from a drift failure. The /casual bypass came from overhead killing small tasks. Each failure leaves an artifact.

The question this raised — for this devlog and for everything else we build — is: how do you formally manage the artifacts? How do you track what enters the pipeline, what each agent sees, and what comes out the other end?

The Board

Our first attempt at formally managing the artifacts was a pipeline — and if you've used a kanban board, you already understand the shape. An item enters on the left, moves through columns, exits on the right:

Karpathy's pattern lives in the INTAKE and BACKLOG columns — markdown files, classification, indexing. The columns to the right are what concurrency and compliance forced us to add. Each column has an entry gate, a WIP constraint (one CR per agent), and a definition of done. The differences are in the enforcement mechanism — our gates are hooks that block, not conventions that suggest.

Where this fails: the pipeline adds overhead. A quick fix that takes five minutes to code takes much longer to govern. We added /casual mode — a bypass that skips the pipeline entirely — because the overhead was killing small tasks. The framework's biggest unsolved problem is knowing when to enforce and when to get out of the way.

What the Pipeline Sees

The kanban board shows the flow. But what happens inside each column? Four intake channels feed the system. The tweet came through one of them:

The tweet came through the Inbox channel — unstructured input, the leftmost column. The subagent classified it: entity agentic-flow-framework, category missing-coverage, severity high. Those three fields are mandatory at creation. They determined everything that happened next.

How Agents See the Knowledge Base

If you work with AI agents, you know this problem: the knowledge base grows, the context window doesn't. Every rule, every spec, every convention you want the agent to follow competes for the same finite space. We hit this wall early. The framework's own governance — rules, schemas, pipeline definitions, agent profiles — consumes context before the agent even starts its actual task.

We measured. The full knowledge base is 615,000+ tokens. An agent's effective context for implementation is maybe 30,000 tokens after the framework overhead. We were seeing context utilization spikes where agents would lose track of their task because the governance material was crowding out the work material.

We're tackling it with three levels of summarization — a way to semantically partition the knowledge base so each agent only sees what it needs:

L3 is one line per entity — enough to decide relevance. L2 is a page of context — enough to orient. L1 is the full spec — enough to implement. The result: 5,105 tokens always on (L3 for all 15 entities plus pipeline rules), and the ability to semantically partition the rest. An agent working on a UI screen pays ~1,900 tokens for its slice of the knowledge base instead of competing with 615,000.

When the tweet became ISS-046, the entity field (agentic-flow-framework) determined which L2 summary the writer agents received. They got the framework's architecture, its blog voice, its previous devlogs. They did not get the ai-sales product specs or the funds module endpoints. The entity classification at intake shaped the knowledge projection downstream.

Is it solved? No. The summaries are generated automatically from the specs, but the compression is lossy — agents sometimes miss details that were in L1 but not in L2. We're struggling with the same context problem everyone building with agents is struggling with. This is how we're tackling it.

But better retrieval doesn't solve everything. Even with the right knowledge base slice available, agents given freedom to explore will re-explore the entire problem space on their own. We saw this repeatedly: an agent with access to the right specs would spend its first 10,000 tokens reading files it didn't need, navigating directories it had no business in, building a mental model from scratch — and by the time it started its actual task, the context was polluted with irrelevant information or missing the critical specs entirely. The decisional freedom that makes agents powerful also makes them wasteful.

The Prompt Assembly

The fix was to remove the exploration entirely. Before an agent writes a single line of code, it already has everything it needs — not because it found it, but because the orchestrator assembled it. The prompt generator resolves the knowledge base into a typed projection:

The checkmarks and crosses are the point. The agent receives its target spec, its entity context, its CR scope. It does not receive other entities' specs, backend specs (if it's a frontend agent), or framework internals. The knowledge base is sliced per task and per role.

The prompt exists as a file on disk. Inspectable. Testable. Reproducible. The agent has never read a single file yet — it starts writing immediately with exactly the knowledge it needs.

The effect is numerical and behavioral. Numerically: agents that used to spend 10,000+ tokens on exploration now start producing code within the first 1,000. Behaviorally: the injected context bounds the agent to its domain. A frontend agent that receives only UI specs, its entity's design profile, and a testing skill reference doesn't drift into backend concerns — not because it's told not to, but because the backend doesn't exist in its world.

Where This Is Going

The current system is Node.js — hooks, scripts, a CLI that reads and writes JSON files. The evolution diagram above ends at the linguistic API. Here's what that looks like concretely — instead of reading rules and deciding what's legal, the orchestrator calls an entry point and the system tells it what's legal:

The guard conditions — [plan exists], [verification passed], [all gates green] — are evaluated by the system, not by the agent. The agent cannot transition to a state whose preconditions aren't met. Not because it's well-behaved, but because the transition doesn't appear in the menu.

The current Node.js implementation works but has limitations — the state machine is hardcoded, the transition graph isn't pluggable, and the hook system is tied to Claude Code's specific lifecycle events. We want to publish it as open source when it stabilizes. The goal is a generic governance layer that any agentic system could use, not just ours. We're not there yet.

By the Numbers

Maybe one day Karpathy's agent will process this devlog through its own knowledge base, classify it, and open an issue. If it does — hi. We started from the same markdown files in a directory. We'd love to compare notes on where they took us.

Session: 2026-04-04 | ISS-046 through ISS-056 | macrocode.ai