TL;DR: A better prompt cannot repair knowledge that was already stale, unreviewed or incorrectly selected before the model saw it. In a small synthetic benchmark, explicit review and version rules improved what reached the model. That still was not enough: a separate application check caught cases where the model cited the current rule and then contradicted it. The practical move is to test knowledge eligibility, version resolution and application consistency as three separate gates.
The answer looked right.
In a synthetic software-release scenario, the assistant said the release was ready for staging. It included the required smoke test. The prose was clear. The checklist looked responsible.
It also carried an old production-route assumption into the final decision: smoke-test the staging route, then treat the production release as complete.
The current rule said something else. Production was not complete until the public route and key assets passed their smoke tests. The context contained that rule, but it also contained an older handover that treated staging as production and a stale record saying a successful deploy command meant the release was finished.
Nothing about the answer looked obviously broken until I asked a different question:
Which facts were allowed to influence it?
That question changes where you look for the defect. The instinct is to rewrite the prompt. Sometimes that is exactly right. But a prompt sits near the end of a longer chain. If stale or conflicting knowledge has already been captured, flattened and admitted to the brief, the prompt cannot restore the state that was removed upstream.
In this test, the prompt was fine. The knowledge lifecycle wasn’t.
The prompt is the last link, not the first
Most discussions about improving AI output begin at generation: stronger instructions, more examples, stricter formatting, another critique pass.
That skips the decisions that determine what the model gets to see.
A knowledge item has a lifecycle. It is captured from somewhere. Someone decides what it means, whether it is current, whether it has been reviewed, what constraints govern its use and whether a newer item replaces it. A retrieval step selects from those items. A brief compresses the selection. Only then does the model generate an answer.
Lose state early and the rest of the system receives a neat-looking distortion:
state is lost
→ stale or unreviewed material remains eligible
→ retrieval rewards textual relevance
→ the brief flattens uncertainty
→ competing claims look equally usable
→ fluent output hides the defectFluency does not create this problem. It makes the problem easier to miss.
The alternative is not to build a bureaucracy around every note. It is to make important knowledge carry enough state to answer a practical question: should this item be allowed to influence this task?
I tested the brief before testing the answer
I built a small synthetic benchmark rather than starting with private client material or a production retrieval stack. The fixture contains 24 task instances, 60 knowledge items and 12 scenarios. Each task was prepared under three conditions:
Raw context: relevant current material mixed with stale or conflicting material.
Flat notes: plain text selected without item-level provenance or lifecycle state.
Structured lifecycle context: items filtered using status, review state, provenance and task constraints.
The first run stopped before model generation. It inspected the context itself.
Four measures matter here:
Precision: how much of the selected context was actually required.
Recall: whether every required item was selected.
Constraint recall: whether the task’s governing rule survived selection.
Leakage: how much stale or conflicting material slipped through.
The structured condition selected every required current item with no stale leakage in this fixture. Precision and recall were both 100%, and every required brief field was present. Raw context retained full recall, but only 40% of what it carried was required and stale or conflicting leakage was 60%. Flat notes were smaller, but recall fell to 87.5% and the source of each claim could no longer be traced reliably.
The first structured rule was still too loose. Requiring only status=confirmed admitted an item that had not passed the necessary review state. I changed one variable: an item now had to be confirmed and either reviewed or corrected.
The improvement happened before the model wrote a word. That is the first useful reframe: better output may begin with a better eligibility decision, not a better instruction.
A reviewed item can still be the wrong version
The review-aware patch exposed another weak assumption.
An imported legacy item can remain both confirmed and reviewed after a newer item replaces it. Review state answers whether an item passed a review rule. It does not answer whether the item is still the version allowed to represent the claim.
I added a stress test with three synthetic scenarios: a refund policy, a release status and a dataset validation split. Each contained one current item, one constraint and 1, 3, 5 or 10 identical copies of an older item. The wording, legacy-first order and top-three retrieval limit stayed fixed. Only the number of old copies changed.
At three or more copies, review-aware retrieval filled 88.9% of its selected slots with legacy records on average. Current-item recall fell to 33.3%, and the governing constraint disappeared.
The fix was to resolve replacement before ranking:
follow explicit
supersedeslinks;group records that represent versions of the same underlying rule;
keep the effective version; and
rank only the survivors.
Resolver results across three synthetic scenarios, exact wording, legacy-first ordering, 3 and 10 legacy copies and a top-three retrieval limit. Image by author.
With complete metadata, explicit replacement links or canonical version selection were each sufficient in the deterministic fixture. Combining them provided redundancy: across 24 variations in wording and item order, the resolver retained the current item and its constraint with no stale leakage. When links contradicted one another or the replacement had not been reviewed, it failed closed instead of guessing.
This does not show that repetition always defeats recency. When the model received all lifecycle metadata in the raw condition, it still chose the current decision. What deteriorated was traceability: with ten legacy copies, it stopped citing the current item altogether.
The bounded finding is narrower and more useful: duplicate legacy records can crowd the current rule and its constraint out of a capped retrieval set when the system ranks before it resolves supersession.
Correct retrieval can still produce the wrong decision
The next failure was more uncomfortable.
I then ran the targeted resolver cases through a local qwen2.5:3b model under fixed settings. In every resolved release case, the combined resolver selected the current release rule and its constraint. The model cited both, then declared production complete even though the cited rule said production remained unconfirmed.
Retrieval had done its job. Application had not.
This is why source citations cannot be the final quality gate. A response can cite the right source and still make a decision that contradicts it.
I froze the same 48 combined-retrieval contexts and applied a fixed checklist. A response had to use the required format, make a decision consistent with the current claim, reject the legacy decision, cite the current source and preserve the constraint. Sixteen responses failed. Each received one revision attempt containing the exact failed criteria, the same selected context and the same model settings.
Fifteen revisions passed. Strict acceptance rose from 66.7% to 97.9%: 15 correction wins, no original wins and one tie. The remaining response corrected its decision and format but still omitted the current source ID.
Application acceptance across the same 48 combined-retrieval contexts before and after one criterion-specific retry. Image by author.
This is not evidence that models generally correct themselves when shown an error. It is a result from one small local model, one synthetic fixture, one locked validator and one tightly specified retry. One failure remains visible.
It does show why the lifecycle needs a third gate. Selecting the current rule is not the same as following it.
Three gates, three different questions
The tests exposed three failure points that are easy to blur into one vague idea called “context quality”. They should remain separate:
1. Eligibility
Should this item be allowed into the task at all?
Check its source, status, review state, freshness, constraints and uncertainty. This gate excludes material that is deprecated, unreviewed, out of scope or otherwise unsafe to use.
2. Version resolution
Is this the effective version of the claim?
Resolve explicit replacement links and canonical versions before ranking. This gate prevents repeated old records from winning through volume or position.
3. Application consistency
Does the answer’s decision agree with the selected claim?
Check the decision, required citation and governing constraint—not merely whether the source was mentioned. This gate catches a fluent answer that quotes the rule and then violates it.
Each gate can fail while the others pass. That is precisely why one overall “quality” score is less useful than several visible checks.
The minimum useful state
You do not need to begin with a vector database, a universal ontology or an elaborate platform.
Start with knowledge items that can answer six questions:
What is this?
Where did it come from?
Is it current and confirmed?
Has it been reviewed?
What constraints govern its use?
What does it supersede, or what supersedes it?
An abbreviated item from the benchmark looks like this:
{
"id": "release-constraint",
"source": "synthetic/release/constraint.md",
"claim": "Production completion requires public-route and key-asset smoke tests.",
"status": "confirmed",
"effective_at": "2026-07-22",
"canonical_key": "release-completion-rule",
"supersedes": ["release-constraint-2024"],
"review_state": "reviewed",
"role": "constraint"
}The implementation can remain ordinary. The benchmark uses standard-library Python to remove explicitly superseded records, retain confirmed and reviewed items, collapse remaining versions by canonical key and only then rank them.
The code, synthetic fixtures and frozen results are available in the public reproducibility repository.
The point is not that this exact schema should become everybody’s schema. The point is that important knowledge should carry enough state to be selected, challenged and corrected deliberately.
What this test does not prove
The fixture is small and synthetic. The runtime used one local qwen2.5:3b model at temperature 0 and seed 0. The checks are mechanical and deliberately narrow. They test source traceability, selected constraints, mutually exclusive decision markers and a fixed response contract—not human preference, broad semantic quality or production outcomes.
The first raw-versus-structured paired result was promising rather than statistically persuasive on its own: structured context passed 75.0% of tasks, raw context 45.8%, with an exact McNemar p-value of 0.092285. The benchmark also does not measure long-term reuse, provider cost or timed human correction effort.
Structure does not make knowledge true. Sources can be wrong. Reviewers can approve a bad interpretation. Dates can be stale. Replacement links can be missing. Two credible sources can conflict in ways that require human judgement rather than filtering.
The value of structure is not certainty. It is visibility. It gives a workflow somewhere specific to fail—and therefore somewhere specific to improve.
Before rewriting the prompt
Inspect the last ten knowledge items supplied to the model.
Can you identify their source, status, review state, constraints, freshness, supersession state and acceptance criteria?
Then inspect the answer. Does its decision agree with the current claim it cites?
If you cannot answer those questions, the next prompt rewrite may only make the same defect sound better.
If you have seen a workflow cite the right source and still make the wrong decision, reply with the failure. That is the edge case worth testing next.
No published comments yet.