We missed one.
Twenty-two questions went into one of our early evaluation runs at Unplugged AI. For twenty-one, the system placed the correct passage within its first three results. The last question described an L-shaped hallway using different language from the source.
The manual called the procedure “L-Shaped Hallway Intersection.” The evaluator asked what the team should do when a hallway turns sharply around a corner. The retriever found the correct manual but surfaced an adjacent passage instead of the section telling the lead Marine to call out the hallway, stop the stack short of the turn, and avoid exposing a body or rifle beyond the corner.
The result came from the right chapter and looked reasonable. It was also wrong enough to matter.
That miss changed how I answer a question we hear constantly: Can offline RAG reach 99% accuracy?
On a narrow retrieval benchmark, probably. But retrieval accuracy alone does not prove that the entire system is 99% reliable.
The Number Everyone Wants
At Unplugged AI, we build AI systems that run entirely on the device in front of the user. The model, documents, retrieval engine, and interface operate without a cloud connection. As CTO, I work within the limits of the phone, laptop, or rugged tablet already in the user’s hands. The model must fit in memory, retrieval must be fast, and answers must remain grounded after we turn the radios off.
Early on, I assumed getting the system to run would be the hardest part. Proving it worked reliably was harder.
“How accurate is it?” is a fair question, but RAG is not one action. The system must interpret the question, find the right document, identify the exact passage, rank it above similar material, and pass that context to a language model without changing the meaning. A failure anywhere in that chain can produce a polished answer that is still wrong.
Our current evaluation contains 61 domain-specific test cases using direct questions, paraphrases, and terminology that does not always match the source. A result counts only when the passage contains the evidence needed to answer the question. Finding the correct document but returning the wrong section is still a miss.
The correct passage currently ranks first 90.2% of the time and appears within the first three results 98.4% of the time. The second number looks better on a slide. The first tells me where we still have work to do.
When the correct passage appears first, the model receives a clear signal. When it appears third beneath two similar passages, it has to resolve ambiguity. That becomes harder with smaller models running locally.
We ran the evaluation on a Pixel 7, in airplane mode with Wi-Fi disabled. Retrieval and generation stayed on the device, with no cloud model, remote search service, or external API. It is still an internal benchmark, not proof of field reliability. Real users will ask questions differently than we do.
What Accuracy Can Hide
A finished demo looks effortless: ask a question, receive an answer, see a citation. Behind it is a lot of staring at nearly identical passages. I adjust ranking, rerun the test, fix one miss, and sometimes push a previously correct result from first place to second.
Then the important questions begin. Did the system preserve the warning? Did it use the source it cited? Did it add a step that was never there? Would the answer still look correct to someone who did not already know the procedure?
Obviously wrong answers are easy to catch. The dangerous ones sound confident, include a citation, and are only slightly off.
Imagine a procedure with eight required steps. The system returns seven but omits the instruction telling the user when to stop and escalate. A simple average may call that nearly perfect. Operationally, it failed.
That is why we separate omissions, contradictions, unsupported additions, ranking errors, and failures to abstain. An invented instruction or missing safety step can block a release.
It is also why the model should not control every part of the system simply because it can. Models are useful for interpreting messy questions, summarizing evidence, and translating formal documentation into natural language. They are less useful when asked to recreate a critical procedure from scratch.
For workflows with mandatory steps or hard stops, we render those elements deterministically. The model can explain the rule and help the user apply it, but it cannot remove or alter the required content. Sometimes the right tool is a model. Sometimes it is retrieval. Sometimes it is a rule that behaves the same way every time.
So, Can Offline RAG Reach 99%?
On a narrow, well-defined retrieval benchmark, probably. We have not yet demonstrated 99% end-to-end accuracy, and a retrieval score alone would not prove that the complete system is 99% reliable.
The percentage matters only when the test behind it is visible. Was the exact passage retrieved or merely the correct document? Was it ranked first or somewhere in the top three? Were the questions written the way people actually speak? Was the final answer checked for completeness and faithfulness? Did the test run on deployment hardware with the network disabled?
Without that context, 99% is mostly decoration.
I would rather show a partner a 98.4% result we can define and investigate than a 99% result built on a convenient test set. One gives us a path toward a better product. The other gives us a nicer headline.
The final test is whether someone outside our team can pick up the device, ask the questions their people actually ask, and expose assumptions we missed. That is why pilots matter.
Reaching 99% once is not the goal. The goal is understanding the system well enough that, when it fails, we know why and we have already decided what it is allowed to do next.
Interested in a free demo or pilot? Reach out to the team at Unplugged AI and we'll show you what your hardware can really do.