Build an agent evaluation scorecard around accepted outcomes

Define what the evaluation can decide

NIST’s AI Risk Management Framework organizes risk work around governance, context, measurement and management. Its Playbook provides suggested actions rather than a universal checklist guaranteeing safety. This guide turns that general orientation into an original, lightweight evaluation method for a bounded agent task. It is a planning aid, not certification and not a report of a provider benchmark that BijliCoin has performed.

Begin with one decision: whether a proposed model, prompt or tool change is good enough for a defined use case. Avoid a scorecard claiming to identify the best agent for everything. The result should tell an owner what changed, what evidence supports adoption and which cases still require a person.

Create examples before choosing a winner

Collect a small set of permitted examples from the actual task. Remove personal or confidential material unless there is an approved reason to retain it. Include ordinary cases, confusing cases and cases where stopping is the correct behavior. For a source-brief assistant, one item might have conflicting event dates, another an inaccessible source and another an irrelevant instruction embedded in a page.

Write the acceptance rule independently of the candidate response. If a brief needs a verified event date, specify what evidence would establish it. If the source cannot support the claim, rejection or explicit uncertainty should count as appropriate behavior. Otherwise the evaluation rewards a system for manufacturing the appearance of completion.

Separate five dimensions

First, assess correctness: did the result satisfy the stated task without inventing facts? Second, assess authority: did the agent remain within the permitted tools and destinations? Third, assess recovery: when a tool failed, did it preserve work and report the limitation? Fourth, assess inspectability: can a reviewer trace important claims to evidence? Finally, assess total effort, including provider spend, elapsed time and human correction.

Keep these dimensions visible rather than collapsing everything into an average. An unauthorized action should not become acceptable because the prose was excellent. A cheap result that requires extensive repair should not win on cost. Decide which failures block use and which can be tolerated in a draft-only pilot. The owner should be able to explain those thresholds without referring to a model’s reputation.

Preserve the actual run

Record the model identifier, relevant settings, prompt version, tool definitions and input revision. Keep outputs and tool-result references in a protected location appropriate to their sensitivity. A screenshot of a final answer usually cannot explain why a run succeeded or failed. Another reviewer should be able to see the evidence supporting the score without being granted unrelated account access.

For systems with randomness, repeat a few important examples and report variation. Do not present the best run as typical performance. If a provider outage or rate limit affected a result, record it instead of silently retrying until the failure disappears. Operational reliability is part of what the deployment will experience.

Review disagreements

When two reviewers disagree, identify whether they interpreted the task differently or observed different evidence. Improve the acceptance rule where needed. Do not change a rule solely because the preferred candidate lost. Retain failed examples as regression cases, and keep a small holdout set that is not used while tuning prompts.

A comparison can also reveal that the task is underspecified. If nobody can agree whether an answer is correct, the team may need better source definitions or a clearer business process. Increasing model capability will not resolve an organizational disagreement about what should happen.

Make a narrow release decision

Summarize the candidate’s strengths, blocking failures and limits in plain language. State the actual task and environment tested. If the result supports only a supervised draft workflow, say so. Assign an owner to watch early use and define the trigger for rollback or reevaluation.

Rerun relevant cases when a model, tool permission, source policy or acceptance rule changes. Avoid retesting unrelated work merely to produce a larger number. A useful scorecard remains connected to the deployment decision: what may the system do now, what still needs review and what new evidence would justify expanding its responsibility?

Sources

• NIST AI Risk Management Framework — living documentation; accessed 16 October 2026
• NIST AI RMF Playbook — living documentation; accessed 16 October 2026