We were measuring the wrong clock.
When I evaluate an offline AI system, I can measure retrieval latency, time to first token, total generation time, and how often the correct passage appears near the top of the results. Those numbers tell me a lot about whether the software is working. They do not tell me whether the person holding the device made a better decision. A system can respond in two seconds and still leave the user spending several minutes deciding whether to trust it. A technically correct answer can arrive too late to affect the outcome. A fast answer that pushes someone toward the wrong action is worse than no improvement at all.
We have started thinking more seriously about a different measurement: operational lift. For us, lift is the difference between how someone performs a task normally and how they perform the same or an equivalent task with AI assistance. The question is no longer only, “Did the system produce the correct answer?” It becomes, “What changed because the system was there?”
The Benchmark Ends Too Early
Traditional AI evaluations usually stop at the model. Did it retrieve the correct passage? Did the response match the reference answer? Did it hallucinate? How long did the generation take?
We need those measurements. Before putting a system in front of a user, we should understand how each part of it behaves and where it fails. But a fielded product is more than a model. It is the model, retrieval engine, source data, interface, hardware, user, environment, and existing workflow operating together. A weakness in any one of those pieces can erase the value created by the others. A retrieval system might find the correct procedure, but the interface could bury the most important warning. The model might provide a faithful summary, but the user could misunderstand what action it recommends. The answer might be correct, but slower to use than opening the original document. In each case, the AI benchmark could look successful while the operational result remains unchanged. Once the answer reaches the screen, there is still an entire human part of the workflow left to measure.
What We Would Measure
The basic comparison is straightforward. First, establish how users perform a defined task using their existing workflow. Then have them perform an equivalent task with the AI system available. Score both against the same predetermined standard. The scenarios cannot be judged by whether the evaluator personally liked the answer. Each one needs a rubric established before the test begins: the acceptable decisions, required steps, critical warnings, escalation conditions, and evidence needed to support the result. From there, three measurements become especially useful.
The first is SOP drift. We use that term to describe the distance between the documented procedure and what actually happens during execution. In a formal evaluation, that can become an SOP deviation rate: the number of required steps missed, performed incorrectly, or completed out of order. That sounds simple until the procedure contains branches. Not every user should follow the same exact path. A condition observed during the scenario may correctly send one person toward escalation and another toward resolution. Before counting deviations, the test has to define which branches are acceptable and what conditions activate them. Otherwise, we risk rewarding people for following a checklist mechanically instead of applying the procedure correctly.
The second measurement is Time to Correct Decision. This is different from response latency. The clock should not stop when the model begins generating. It should stop when the user reaches and commits to the correct decision. The AI may retrieve information in milliseconds and generate an answer in seconds, but those are only parts of the workflow. The user still has to read the response, interpret it, compare it against the situation, and decide what to do. A faster wrong decision is not lift. For that reason, speed should only count when the final decision is correct. In some scenarios, the correct decision may be to stop, escalate, request support, or state that the available information is insufficient. Reaching that conclusion quickly can be more valuable than producing a confident answer. We would also look beyond the average. A strong median time can hide a small number of serious delays. In field operations, those worst cases may tell us more than the average does.
The third measurement is Missed Critical Steps. Not every deviation carries the same consequence. Missing a minor documentation field should not be scored the same way as ignoring a hard stop, skipping a safety warning, or failing to escalate when required. A system could reduce several low-consequence errors while doing nothing to prevent the mistake that matters most. Critical steps have to be identified before the evaluation and reported separately. They should not disappear inside a general accuracy percentage. That is also one reason we do not believe every part of a procedure should be generated freely by a language model. Mandatory instructions, hard stops, and required acknowledgments can be rendered deterministically. The model can help interpret the situation and explain the source, but it should not be allowed to quietly remove a required step. Making a warning appear every time is one problem we can solve in software. Whether the person actually catches it, understands it, and changes their behavior is a different problem.
What Lift Would Actually Prove
Lift is not one universal score. A system might reduce Time to Correct Decision without changing the critical-step miss rate. That could mean it improves information access but does not yet improve procedural execution. Another system might reduce SOP deviations while adding time. That tradeoff could be acceptable in a high-consequence workflow and unacceptable in one where every second matters. The average performance might improve while a few users perform significantly worse. That could point to unclear interface design, overreliance on the recommendation, or a system that works well only for people with a certain level of experience. None of those results fit neatly into a single “AI improved performance by X percent” headline, and that is probably a good thing. They give us a much clearer picture of what the system is actually helping with and where it still falls short. Early pilots will not prove that an AI system improves every operator, mission, or environment. The scenarios will still be limited. Participants will know they are being evaluated. Repeating similar tasks can create learning effects. Real field conditions will introduce pressures that controlled testing cannot fully reproduce. The goal is not to manufacture one impressive percentage. We want to make a narrow claim, define what would prove or disprove it, and test against that. I still care about retrieval accuracy, response faithfulness, latency, memory use, and model behavior. We need those numbers to know whether the technology itself is working well enough to put in front of people. But once it is, the more interesting question is what happens to the person using it.
Can they reach the correct decision sooner? Do they follow the applicable procedure more closely? Do they miss fewer of the steps that actually matter?
Accuracy tells us whether the system found the answer. Lift tells us whether the answer changed what happened next.
We built Unplugged AI for the moment the signal drops and the decision can't wait. If that's a moment your people know well, we'd like to hear from you.