Evaluating retrieval before evaluating answer style

Inspect evidence before rewriting the answer prompt

Separate retrieval evaluation from response evaluation. For each test question, identify the source material that should support a correct answer. Check whether it appears among the retrieved candidates and whether enough context survives assembly. A better writing prompt cannot reliably repair evidence that was never supplied. This separation makes failures easier to locate.

Worked example

Illustrative scenario; not a claim of measured client results.

Suppose an assistant gives the wrong warranty period. First inspect whether the current warranty policy was retrieved. If an obsolete policy was selected, fix source authority or filtering. If the correct passage was supplied but misinterpreted, investigate the answer stage. If no approved policy exists, the appropriate outcome may be an explicit gap rather than a more confident response.

Checks, limitations and next decisions

Track failure categories such as missing source, wrong candidate, lost qualifier and unsupported synthesis. Use examples from each category when testing changes. Avoid an aggregate quality score that hides whether the system is retrieving better evidence or merely producing more persuasive wording. A reliable improvement should be visible in the evidence path as well as the final answer.

A practical starting point

Work through these checks in order. Use the results to decide what needs to change in your own workflow.

1. Build a set of questions with known supporting documents.
2. Inspect the retrieved passages without generating an answer.
3. Fix missing coverage, ranking or filtering problems before tuning presentation.

A mistake to avoid

Do not use fluency as a proxy for factual support.

How to check progress

Track retrieval relevance and answer correctness in separate columns.