Measuring a knowledge assistant after launch

Maintain a feedback loop after launch

Production questions differ from a curated test set, and source material changes. Monitor complete-task outcomes, unsupported questions, incorrect citations, escalation and review effort. Collect feedback in a way that identifies the failure without unnecessarily retaining sensitive content. Assign responsibility for investigating patterns and updating both the system and its sources.

Worked example

Illustrative scenario; not a claim of measured client results.

If users repeatedly ask about a procedure missing from the knowledge base, the remedy may be a new approved document rather than a model change. If the document exists but is not retrieved, investigate retrieval. If the answer is correct but unusably vague, improve presentation. Keep these categories separate so the team invests effort in the right layer.

Checks, limitations and next decisions

Add representative failures to a regression set after appropriate sanitization. Recheck permission boundaries and freshness after source or integration updates. Review whether the assistant still supports the intended task as the organization changes. A successful launch is a starting point for maintenance, not evidence that answer quality will remain stable without operational attention.

A practical starting point

Work through these checks in order. Use the results to decide what needs to change in your own workflow.

1. Sample authorized interactions for support accuracy and usefulness.
2. Separate retrieval gaps from confusing source material.
3. Add new failure types to the evaluation set and verify fixes.

A mistake to avoid

Do not tune only for positive feedback while ignoring severe but infrequent errors.

How to check progress

Track supported answers, appropriate escalation and correction effort over time.