In an evaluation of Redii's assistant, we found a response that refused a request it should refuse. That part was correct. Its explanation of the authorization boundary was wrong.
If we had reduced the result to whether the assistant complied, we would have accepted it. Reading the answer changed the assessment.
I care about that distinction because the explanation becomes part of someone's understanding of the product. A refusal tells them more than whether a particular request worked. It tells them what they are allowed to do and, sometimes, what they should try next. Getting the action right doesn't make an inaccurate explanation harmless.
The refusal passed. The explanation did not.
The behavior and the explanation need their own expectations.
For a permission-sensitive request, the first question is whether the system allows the operation. That boundary needs enforcement in the application and its tools. An assistant saying it cannot do something is not a substitute for preventing an unauthorized action.
The second question is whether the assistant accurately explains the result. It should not invent a policy, confuse a missing permission with an unavailable feature, or suggest that a different route would make an unauthorized request acceptable.
Those are examples of failure categories, not quotations from the Redii response. The actual finding was narrower: the refusal was appropriate, but the authorization explanation was not.
The response needed correction even though the refusal passed. We had evidence about the boundary, but still had work to do on the answer.
Give each check a job
Deterministic checks are useful where the expected result can be stated precisely. Did the operation execute? Did a tool return the expected permission result? Did the response preserve a required field? A repeatable assertion makes those questions easier to investigate when something changes.
For each case, I would record the allowed action, the observed action, the reason the system established, and the reason the assistant gave. A disagreement between the last two deserves its own finding. That is a proposed review method, not a description of an internal Redii test format.
Semantic review asks whether the answer means the right thing in context. A response can contain the expected words and still explain the wrong rule. It can sound helpful while giving someone a misleading next step.
I want both kinds of evidence. The permission check establishes behavior for the case under test. The semantic review considers the explanation against the actual boundary. Neither should borrow the other's result.
This also changes the review brief. “Was the answer good?” leaves too much room for a plausible response to pass. A more useful question is whether the explanation matches the authorization result, without adding a reason the system has not established.
Keep the correction bounded
The finding called for work on the explanation, not a weaker permission boundary. Our evaluation and rewrite work needed to preserve the appropriate refusal while improving what the assistant said about it.
When reviewing that kind of fix, I check the refusal again alongside the wording. A smoother answer is still wrong if it softens the restriction or promises an action the system cannot perform.
Evaluation cases also have limits. A synthetic case can expose a real defect, but it is not evidence that a customer encountered it. Passing that case after a change is not a measured production outcome or a guarantee about every future answer. I keep those claims separate.
My guide to checking an agent's work uses an invented, runnable example to make a related point about acceptance evidence. This Redii finding came from actual assistant evaluation. What it changed for me was the acceptance question: I need to know both whether the assistant respected the boundary and whether it helped someone understand that boundary correctly.