PART II — VERIFICATION: THE FOUNDATION OF TRUST

CHAPTER 4

The Verification Problem

The mistake began with a number.

It was a small number, printed in the middle of a report Daniel had been reading on his phone. The report claimed that a particular market was growing rapidly and that businesses entering early could benefit from the expansion.

The figure was specific. The explanation was detailed. Several articles appeared to support the same conclusion.

Daniel had seen claims like this before. Most of the time, he would have noted the figure, considered its implications, and moved on to the next question.

This time, he stopped.

He had been building his council for weeks, assigning responsibilities to different AI roles and refining the process through which they investigated a decision. He had learned to question assumptions, invite disagreement, and separate recommendations from evidence.

But the number in front of him exposed a weakness he had not fully addressed.

What if the information that entered the council was wrong?

The Investigator could find a report containing a false figure. The Analyst could use that figure to calculate a convincing projection. The Challenger could identify weaknesses in the business model while accepting the market estimate as accurate. The Synthesiser could organise all their findings into a clear, professional conclusion.

Every stage might appear to work.

The final decision could still rest on a false premise.

Daniel stared at the number.

He had been concentrating on how to make the council reason better.

Now he had to examine how the council established what was real.

The Number Everyone Repeated

Daniel began with the original report.

He wanted to know where the market estimate had come from, how it had been calculated, and what the report actually meant by the word market.

The report cited another publication. That publication referred to an industry analysis. The industry analysis mentioned a dataset compiled by a research organisation.

The number had travelled through several documents, acquiring credibility each time someone repeated it.

Daniel followed the trail.

At first, the process seemed promising. The publications looked different, and the explanations were written in different styles. Some discussed the market in general terms. Others used the figure to make arguments about future growth.

But the trail kept returning to the same underlying estimate.

There were not several independent measurements.

There was one estimate being repeated by several sources.

That did not automatically make the figure wrong. A reliable original study could legitimately be cited by many publications. But the number of articles repeating the estimate could not tell Daniel whether the original calculation was sound.

He needed to examine the source itself.

When he found the underlying report, he discovered that the figure referred to a particular segment of the market, not the entire market he had assumed it described.

The difference was significant.

The estimate might still be useful, but it could not support the broader conclusion without additional evidence.

Daniel returned to the AI assistant that had summarised the original report.

It had not necessarily invented the number. It had reproduced information that appeared in the material available to it. The failure had occurred when the number was interpreted more broadly than the underlying evidence justified.

The assistant's answer had been useful as a starting point.

It had not completed the verification.

Daniel added a note to his working document:

Before trusting a claim, establish exactly what the claim says, where it originated, and what the evidence is capable of proving.

Then he added another sentence beneath it.

Do not confuse the popularity of a claim with the strength of its evidence.

When an Answer Sounds Like Knowledge

The discovery made Daniel reconsider some of his earlier conversations with AI.

He remembered answers that had arrived with impressive detail. They contained dates, explanations, comparisons, and references to events he had not known about. At the time, he had sometimes treated that detail as evidence that the assistant had understood the subject thoroughly.

Now he could see the problem.

A system could generate a coherent explanation without having established every factual claim inside it. It could misunderstand a source, combine incompatible information, or present an uncertain detail as though it were settled.

A response could be sincere in tone and still be wrong.

The danger was not limited to obviously absurd answers. Those were often easy to reject.

The more difficult errors were plausible.

They fitted the subject. They used familiar terminology. They sounded like things that could reasonably be true.

And because the explanation was easy to understand, the reader could move directly from reading it to believing it without noticing that an important step had been skipped.

Daniel began distinguishing three things that he had previously allowed to blur together.

An answer could be plausible because it made sense.

It could be supported because relevant evidence existed.

It could be verified to a reasonable standard because the important claims had been checked against appropriate sources.

These categories overlapped, but they were not interchangeable.

A plausible explanation deserved consideration.

A supported explanation deserved examination of its supporting evidence.

A verified claim deserved confidence proportionate to the quality and independence of the checks performed.

Even verification had limits. A document could be authentic but outdated. A dataset could be accurate but irrelevant to the question. Several independent sources could still share a common limitation.

The goal was not to declare a claim permanently immune to doubt.

It was to establish whether the available evidence justified relying on it for the decision at hand.

Daniel realised that this was the missing foundation of his council.

Without verification, the council could become a sophisticated machine for organising mistakes.

A Test of the Council

The following week, Daniel gave his council a problem.

He presented a business proposal containing several factual claims, including an estimate of customer demand, a description of the competition, and a projection of potential revenue.

He asked each role to perform its assigned function.

The Investigator gathered the available information and traced the proposal's main claims to their sources.

The Analyst examined the financial structure and identified the assumptions connecting customer demand to revenue.

The Challenger searched for weaknesses in the proposal and considered alternative explanations for the apparent opportunity.

The Verifier examined the most important claims against evidence outside the original presentation.

The Synthesiser would organise the findings once the investigation was complete.

The process revealed a problem almost immediately.

The proposal claimed that a large number of potential customers were underserved. But the evidence supporting the claim came from a survey conducted among people who had already expressed interest in the product.

The survey could tell Daniel something about those respondents.

It could not, by itself, establish the level of demand across the broader population.

The distinction changed the analysis.

The projected customer base was no longer a verified starting point. It was an assumption that needed further testing.

Daniel asked the Analyst to recalculate the projections under several scenarios. What would happen if only a small fraction of the interested respondents became paying customers? What if acquisition costs were higher than expected? What if customers tried the product but did not return?

The figures changed.

The business still had possibilities, but the original projection no longer deserved to be treated as the expected outcome.

Daniel looked at the results.

The council had not discovered a secret hidden inside the proposal. It had identified a gap between what the evidence established and what the proposal claimed.

That gap was enough to change the decision.

He had been close to evaluating the opportunity as though the projected demand were a fact. The verification process showed that the demand had not yet been established.

Before investing heavily, he needed evidence from a broader group of potential customers and, preferably, actual purchasing behaviour.

The lesson was simple.

A calculation could be mathematically correct while producing a misleading conclusion because its starting assumptions were weak.

The Analyst could calculate accurately.

The Verifier had to examine whether the important inputs deserved to be trusted.

The Limits of the Verifier

Daniel was tempted to believe that he had solved the problem by creating a dedicated verification role.

Then he caught himself.

Who was verifying the Verifier?

If the Verifier used an AI assistant to inspect a claim, it could make the same mistakes as the other roles. It could misread a document, overlook a qualification, or cite a source that did not support the conclusion.

Giving a role the name Verifier did not grant it special access to truth.

The responsibility had to be defined by the work performed, not by the title assigned to it.

Daniel revised his instructions.

The Verifier could not simply return a confident statement that a claim was correct. It had to identify the claim, locate the supporting evidence, explain how that evidence related to the claim, and disclose any important limitation.

For consequential claims, Daniel wanted to see the original document whenever possible.

If a claim depended on a calculation, he wanted the inputs and method available for examination.

If a source was secondary, he wanted to know whether a primary source existed.

If two sources disagreed, he wanted the disagreement reported rather than silently resolved in favour of whichever answer sounded better.

And if a claim could not be checked, the Verifier had to say so.

The process could be inconvenient.

It was also more honest.

Daniel knew that he could not personally reproduce every specialised investigation. Some questions required expertise, access to restricted records, laboratory testing, legal interpretation, or professional judgement beyond his competence.

Verification was not a promise that one person could check everything alone.

It was a discipline for understanding what had actually been checked, by whom, against what evidence, and with what limitations.

The more consequential the decision, the more important those questions became.

The Evidence That Refused to Agree

A few days later, Daniel encountered a different problem.

He was examining a company whose public statements described strong growth. One report praised its expansion. Another questioned the sustainability of its financial performance.

Both cited figures.

Both appeared credible.

And both seemed to support different conclusions.

Daniel's first instinct was to search for another source that could settle the disagreement.

He found more articles, but they did not resolve the issue. Several repeated the company's public statements. Others relied on the same financial reports while emphasising different measures.

The search produced more material without producing a clearer answer.

Daniel stopped and examined the disagreement itself.

The sources were not necessarily contradicting one another on every fact. One was discussing revenue growth. Another was discussing profitability and operating costs.

A company could increase revenue while losing money. It could expand its customer base while struggling to cover its expenses. It could report a promising quarter without having established long-term sustainability.

The apparent contradiction had partly arisen because the sources were answering different questions.

Daniel separated the claims.

Was revenue increasing?

Were costs rising at the same rate?

Was the company generating positive operating cash flow?

How much of the reported growth was recurring?

What evidence supported expectations about future performance?

Some of these questions could be examined using published financial statements. Others required more information or remained dependent on future events.

Daniel's confidence did not increase simply because he had found a larger collection of reports.

It increased where the evidence became clearer.

Where the evidence remained incomplete, he left the uncertainty visible.

That was a difficult habit to develop. People often preferred a single explanation, especially when competing explanations made action feel complicated.

But forcing a conclusion did not eliminate uncertainty.

It only concealed it.

Daniel began to understand that a reliable council needed the courage to return an incomplete answer when the evidence was incomplete.

What Counts as Independent?

The next weakness appeared during a discussion with Marcus.

Marcus had found several articles supporting a business decision he wanted to make. He showed Daniel the results on his phone.

“Look,” Marcus said. “There are multiple sources saying the same thing.”

Daniel examined the articles.

They were published by different websites, but several referred to the same original announcement. One quoted another article, which in turn cited the announcement. Another summarised a report that repeated the same figures.

The sources were different.

The evidence beneath them was not necessarily independent.

Marcus frowned.

“So how many sources do I need?”

Daniel shook his head.

“That's not the first question.”

“What is?”

“What would count as independent evidence for this particular claim?”

The answer depended on the claim.

If they wanted to establish that a company had announced a new product, the company's original announcement might establish that the announcement had been made. It would not necessarily establish that the product performed as promised.

If they wanted to establish that a product worked, they would need evidence relevant to its performance, not simply more articles repeating the company's description.

If they wanted to know whether customers would pay for a service, several market reports might provide useful context, but actual purchasing behaviour could offer more direct evidence.

The standard had to match the question.

Daniel explained that independent evidence did not always mean a different website or a different AI system. It meant examining whether the evidence genuinely provided a separate basis for the conclusion.

Two reports using the same dataset might offer different interpretations, but they would not represent two independent measurements of the underlying phenomenon.

Three AI assistants given the same unverified claim might generate three explanations, but their agreement would not independently establish the claim.

The number of voices mattered less than the relationship between their evidence.

Marcus looked at the articles again.

He had been counting sources.

Now he was examining what those sources actually contributed.

The Cost of Getting It Wrong

As Daniel refined the verification process, he began applying different levels of scrutiny to different decisions.

He did not investigate every ordinary question as if he were preparing a legal case. That would make the system so slow and demanding that he would eventually stop using it.

Instead, he considered the consequences of error.

If he wanted to understand a general concept, an AI explanation could be a useful starting point. If the explanation contained unfamiliar terms or important factual claims, he could check those details as needed.

If he wanted to publish a factual statement, he needed to be more careful about its accuracy, context, and source.

If he was considering committing a substantial portion of his savings, the burden of verification became heavier.

If a decision affected someone's safety, rights, health, or legal position, he needed appropriate evidence and, where necessary, qualified professional advice rather than relying on AI alone.

The principle was not that important decisions always required certainty. Certainty was often unavailable.

The principle was that the effort devoted to checking a claim should reflect both the importance of the claim and the consequences of being wrong.

Daniel wrote a question at the top of the verification section in his notes:

What could this mistake cost?

He found it more useful than simply asking whether an answer looked reliable.

The question forced him to consider the decision that would follow from the answer.

A small factual error in an informal conversation might have little consequence. The same error, embedded in a financial projection or a public claim, could affect many people.

The answer could not be evaluated separately from its intended use.

Verification was not merely about whether a statement was true.

It was also about whether there was enough reliable evidence to justify acting on it.

The Question That Changed the System

One evening, Daniel reviewed the council's latest findings.

The system was becoming more disciplined. Its roles were clearer, its questions more precise, and its conclusions more careful about distinguishing facts from assumptions.

Yet Daniel knew that no process could eliminate every error.

Sources could be incomplete. Records could be outdated. A calculation could contain an unnoticed mistake. Independent investigators could reach different conclusions because the available evidence supported more than one interpretation.

The council could not promise that every claim would be verified.

It could, however, stop pretending that every claim had been verified.

That was an important difference.

He added a status field to his working notes. Important claims would be marked according to their actual state: verified to the available standard, supported but not independently confirmed, disputed, unverified, or contradicted by the evidence examined.

The labels were not permanent verdicts. New information could change them.

Their purpose was to prevent uncertainty from disappearing as a report moved from one stage to the next.

If the Investigator marked a claim as unverified, the Analyst could not quietly treat it as established fact without explaining the assumption.

If the Verifier found conflicting evidence, the Synthesiser had to preserve that conflict in the final assessment.

If a crucial question remained unanswered, Daniel would see it before deciding whether to proceed.

The system was becoming less interested in producing a smooth conclusion and more interested in preserving the integrity of the reasoning that led to it.

Daniel thought about the original market estimate that had started the investigation.

A single number had moved through several publications, been repeated by multiple systems, and appeared more established than it really was.

The problem had not been a lack of intelligence.

It had been a failure to distinguish repetition from evidence, interpretation from fact, and confidence from verification.

He saved the revised protocol.

For the first time, he felt that the council was developing a foundation strong enough to support more difficult work.

But one problem remained.

Even when he found a credible source, he still had to decide whether to trust what that source was telling him. A source could be authentic and still be biased. An expert could be knowledgeable and still be mistaken. A report could contain accurate information while leaving out the detail most relevant to the decision.

Verification could establish where a claim came from and whether the evidence supported it.

It could not remove the need to examine the relationship between the source, the claim, and the person interpreting both.

Daniel closed his notes.

The next question was no longer simply whether an answer could be checked.

It was how to distinguish a source that helped him see reality more clearly from one that merely reflected what he already wanted to believe.

He had learned to demand evidence from the answer. Now he needed to learn how to examine the source behind the evidence.