Skip to content
Digital Souls
Contact

19th November 2026 · 15 min

Proof of Concept Testing: How to Run One That Decides

Proof of Concept Testing: How to Run One That Decides

The standard advice on proof of concept testing goes like this: build the smallest thing that demonstrates the idea, show it to a handful of people, and once it works, you have permission to start the full build. It sounds disciplined. In practice, it produces expensive theatre: a demo running on curated data, in a room, with the person who built it driving.

A demo proves that software can be made to do something once. It doesn't tell you whether the approach survives messy inputs, whether the people who must use it will keep using it, whether it clears security and procurement review, or whether the savings cover the running cost. Those are the questions that decide whether a project should exist at all.

So treat a proof of concept as a decision experiment. Write the question it must answer, fix the pass mark, decide what each result triggers, and name the person who signs off. All of that happens before anyone opens an editor.

Table of Contents

What a Proof of Concept Test Actually Proves

A proof of concept tests whether a core idea works under controlled conditions. It doesn't test whether a finished product is ready for customers, and it can't tell you what running that product will cost.

The UK Procurement for Innovation Playbook sets that boundary in a national framework. It classifies experimental proof of concept as Technology Readiness Level 3, sitting after basic principles observed at TRL 1 and technology concept formulation at TRL 2. Laboratory validation follows at TRL 4, then validation in a relevant environment at TRL 5, demonstration in a relevant environment at TRL 6, and operational demonstration at TRL 7.

An infographic titled What a Proof of Concept Test Actually Proves with four numbered stages explaining testing.

Think of a scale model of a bridge. Engineers load it until something bends, and the model shows the design can carry its own weight. Nobody opens the bridge to traffic on that evidence. Traffic, weather, inspection regimes and maintenance crews come much later, and each one asks a different question.

The same playbook describes what follows a proof of concept as alpha work, which means building and testing several prototypes with small user groups to identify the most viable option. For digital products that translates into three jobs in sequence. Test technical feasibility first. Test usability and user value second. Validate performance in realistic operating conditions third.

Stage Question it answers Typical setting Decision it informs
Proof of concept (TRL 3) Can the core idea work at all? Small team, controlled conditions Whether to fund prototype work
Laboratory validation (TRL 4) Do the components work together? Lab or sandbox Whether to test in realistic conditions
Relevant-environment validation (TRL 5-6) Does it hold with realistic data and users? Live-like environment Whether to run a pilot
Operational demonstration (TRL 7) Does it run reliably in day-to-day use? Production-like service Whether to scale

By the end of a well-run test you should be able to answer three things in one sentence each. Can the idea work at all, meaning the mechanism behaves the way the theory says it will. Does it work with real inputs, meaning messy production data rather than the tidy samples used in the first pass. Does anyone need it, meaning the people you built it for change what they do as a result. A proof of concept that answers only the first is a science experiment with a budget attached.

How to Set Success Criteria Before You Build

Criteria chosen after you see the results invite wishful reading. Every founder can find a reason a 54% completion rate is basically fine once the build is already paid for, and every steering group will find a different reason. Setting the marks first removes that argument before it starts.

  1. Write the single question the test must answer. One question, in one sentence, such as whether an AI agent can resolve routine support tickets without a human. Teams that write three questions usually answer none of them, because the test design gets pulled in three directions and satisfies nobody.
  2. Name the highest-risk assumption behind the idea. Risk, not feature list, decides what you build. The common slip is testing the parts you already know how to build, which produces a comfortable result and teaches you nothing about the part that could sink the project.
  3. Choose two to four measurable outcomes that answer the question. Task-completion rate, error frequency, time per task, support contacts generated. Two is often enough. Measuring everything is a symptom of not having decided what matters.
  4. Set a pass mark for each outcome and list the constraints that must also hold. A minimum completion rate, a maximum error rate, plus the data-protection and integration limits the solution has to respect. "Improve efficiency" is not a pass mark, because nobody can fail it.
  5. Decide in advance what each result triggers. Stop, redesign, continue or scale. Written next to the pass mark, this turns a discussion into an instruction.
  6. Name the owner who signs off. One person, by role, with the authority to release or withhold the next budget. A test with no owner produces a report that nobody acts on.

Writing the pass marks in the same plain language as a software requirements specification example keeps the test and the eventual build aligned. If the criteria and the specification drift apart, you end up testing something you won't build.

Practical rule: if the pass mark doesn't fit on a single page, the test isn't ready to start.

A flowchart showing six numbered steps to establish success criteria before starting product development.

Testing the Assumption That Could Sink the Idea

A proof of concept should attack the riskiest causal assumption, not the whole product. The Magenta Book guidance on test and learn makes this the core discipline: test critical assumptions in realistic or operational settings, run controlled and time-limited trials, and combine qualitative evidence with quantitative measures of usability, feasibility, unintended effects and early outcomes.

Take a SaaS team that wants an AI tool to triage support email. Three assumptions sit behind that idea. The classifier routes messy real enquiries correctly, support staff trust the suggestions enough to act on them, and the historical ticket data is clean enough to learn from. Of those, classification accuracy on real email is the one that decides whether the rest matters.

So the test isolates it. Sample around 200 real, anonymised emails, run them through the classifier, and record three things per email: whether it routed correctly, how long the routing took, and what type of failure occurred when it didn't.

Separate model errors from data errors

Instrument the test so every decision is logged, every failure carries a reason tag, and each tag is classified as a model problem or a data problem. A classifier that routes wrongly because the enquiry was ambiguous is a different finding from one that routes wrongly because the training labels were inconsistent. The first may be acceptable; the second means you have a data project, not an AI project, and the budget looks nothing like the one you planned.

This is the foundation-before-walls principle. Nobody debates the windows while the footings are still untested, because one load-bearing assumption decides whether everything above it is worth building.

A test that cannot fail is a demonstration, not a proof of concept.

One caution the guidance is explicit about: small-scale testing isn't designed to prove final effectiveness. Claims of causal impact at programme scale need stronger evaluation, including counterfactual methods where those are feasible. A clean result means the assumption held in your test. It doesn't mean the outcome would hold everywhere.

Designing a Small, Time-Limited Test

Scope decides cost, and cost decides whether the experiment happens at all. Hold the test to one workflow, a small group of relevant users and a fixed window, typically two to six weeks. A narrow, short test keeps the spend low and forces a decision date into the calendar, which is the part that most internal projects never get.

  • Data: use anonymised real samples wherever data protection allows. Synthetic data flatters results, because it contains none of the mess that production will deliver within a week of launch.
  • People: involve the staff who will run the system, not just the sponsor. Their workarounds are your most valuable finding.
  • Governance: bring compliance and security in at the start. A technically successful test that fails a security review has cost you the full budget and returned nothing usable.
  • Money: set a ceiling before you begin and write it down. A test that can expand to absorb whatever budget exists will do exactly that.
  • Out of scope: list what you're deliberately not testing and repeat that list in the report. This is the only reliable defence against overclaiming later.

Say a retailer wants to know whether a returns chatbot reduces agent workload. It runs the bot with a dozen agents for two weeks on one product category, with a fixed budget, a named reviewer from operations and a support lead who owns the queue. That scope produces an answer about one category and one team, which is enough to decide whether a wider trial is worth funding. The same project run across all categories with everyone involved produces a bill and no clean signal.

A useful parallel is how you scope a first release: build a minimum viable product around the smallest version that answers a real question, not the smallest version that demos well.

State the limit in plain words at the top of the report. A small test shows feasibility and early behaviour. It is not proof of impact at full scale, and any document that implies otherwise will be quoted back at you in a board meeting eighteen months later.

Reading Results Across the Full Funnel

Averages hide failures. The evaluation of the Employment and Health Discussion proof of concept shows why. The service began in a single Jobcentre Plus location to test delivery feasibility, then expanded to 13 offices once early learning had changed referral processes, clinical governance and staff training. During the evaluated period, 1,219 eligible referrals were recorded, but only 333, or 27%, completed the process. Satisfaction among participating customers was nevertheless 95%.

Read those two numbers together. People who got through were happy. Many never got through. High satisfaction did not compensate for a completion bottleneck, and a dashboard that reported only the satisfaction score would have looked like a success.

Measure What it reveals Where it usually breaks
Eligible-to-completed rate Whether the full workflow finishes Hand-offs and onboarding
Task-completion rate Whether users finish the core task Unclear steps and usability
Error frequency Reliability of outputs Data quality and edge cases
Latency Speed under realistic load Integrations and infrastructure
User satisfaction Whether people value the result Can hide throughput problems
Support workload Hidden operational cost Unclear ownership after launch

Diagnosing a drop-off

Work backwards from the final number and locate each fall. Segment by site, by user type, by workflow and by failure reason, then classify the cause. A drop at onboarding is usually usability. A drop at the point where two systems exchange data is integration. A drop that varies wildly between sites is operational readiness, and a drop that clusters around one team is training. A fall in the number of people arriving at all is demand generation, which is a marketing problem wearing a product costume.

Each figure goes back to the pass mark you set before the build. Not against a competitor, not against a feeling, and not against a revised target invented after the results landed. The comparison takes minutes if the criteria were written properly, and it can't be resolved at all if they weren't.

Planning Adoption Before You Commit to a Build

The gap that stops successful tests isn't technical. A 2025 Public Accounts Committee report found that 70% of surveyed government bodies were piloting or planning AI, but only 37% had deployed it, and it noted the absence of a systematic mechanism for sharing learning between pilots. Experimentation isn't the bottleneck. The transition from isolated trial to governed production service is.

The same report shows why adoption belongs inside the test rather than after it. A cross-government experiment involving 20,000 people reached 83% adoption in its first month and reported average savings of 26 minutes per day, with confidence and familiarity with the tool correlating with the time saved.

Measure What it reveals Where it usually breaks
Adoption rate Share of eligible staff using the tool in the first month, as in the 20,000-person experiment Staff quietly bypass the tool
Time saved Average minutes saved per person per day, measured in the same experiment Benefit is never verified after launch
Confidence and familiarity Whether training is converting into time savings Training gaps and low trust
Usage persistence Whether use survives past the novelty of week three Early enthusiasm fades after the pilot
Performance by role Whether some roles gain far more than others Averages hide role-level gaps

Track five things in the human layer: who uses the system, whether use persists, how performance differs by role, what training each group needs, and how the benefit gets verified after launch. Change management, data governance and support ownership are design items during the test, not details to be settled once the thing is live.

What to instrument: log active users per role and per week, so a slow drift away from the tool shows up as a number rather than an anecdote.

A customer-support agent can work perfectly and still fail. Staff stop opening it because it adds a step to their day, the queue keeps moving without it, and nobody records the drop because adoption was never measured. A test that tracked weekly active users by team would have caught that in week three, while a fix was still cheap.

Deciding to Stop, Redesign, Pilot or Scale

A proof of concept should end with a decision, not just a slide deck. Its result is not a verdict on the technology as a whole. It shows whether the evidence supports stopping, redesigning, piloting or scaling the idea.

A person standing at a crossroads deciding between stopping, redesigning, piloting, or scaling a project.

Outcome Evidence required Next action
Stop The core assumption fails in realistic conditions, and no affordable fix is evident Record the findings and close the test
Redesign The test partly succeeds, with a clear cause that can be changed Change one variable and retest
Pilot The criteria are met in an environment relevant to real use Define the pilot's scope, users and owner
Scale The criteria hold across segments, and post-launch measures remain stable Plan rollout and assign support ownership

When a technical pass isn't a go

A system can work as designed and still be a poor investment. UK evidence on barriers and enablers to advanced technology adoption describes how decisions to adopt advanced technology are shaped by the clarity of the use case, affordability, regulation, risk and trust. Smaller firms may also need practical support to test tools when internal resources are limited. If a proof of concept passes technically but cannot clear procurement, security or workflow requirements, the result is not a go.

Set decision thresholds before building. Specify the time savings needed, acceptable error rate, data-protection limits, integration effort and payback period. Name who would own implementation after the test. These conditions turn a technical result into an investment decision. They also prevent teams from changing the definition of success at the readout.

A decision record works like a map for the next team: it shows why the project stopped or continued. Keep it to one page, with the chosen outcome, supporting evidence, money spent and owner of the next step. The record should be concise, but specific enough to explain the reasoning.

For a larger programme, a product development roadmap can show where the test fits, what decision it informs and who owns the next stage. The roadmap helps connect a small experiment to the wider delivery sequence without treating a successful test as automatic approval to build everything.

The video below offers a companion explanation.

Before committing build budget, put the pass criteria, constraints and four possible outcomes in a short test plan. Share it with the budget owner and compliance lead so procurement and compliance requirements are considered before the decision, not after.

Digital Souls Studios LTD works from a written specification through production, launch and support. If you are planning a proof of concept for a SaaS product, an AI agent or an internal tool, visit Digital Souls Studios LTD and tell us what your test has to answer.