Build a local evaluation set for an AI agent

Start with the evidence the score should represent

A green score can be meaningless when the expected answer was chosen after reading the candidate’s response. This workflow demonstrates a small evaluation harness whose cases already contain expected outcomes. It uses an ordinary local function as the candidate. No model is called. That choice makes it easy to inspect the evaluator itself before using it to judge a less predictable system. The output is a case-by-case comparison and a count, not a claim that an AI agent has passed a benchmark.

Define the synthetic task

The fixture decides whether a source brief can advance to draft or needs review. Its input has a source count, a Boolean indicating whether a date was checked, and a Boolean indicating a conflict. The baseline returns draft only when its narrow rule is satisfied. Those inputs are declarations supplied by the fixture, not independently verified facts. Two sources could be unreliable, and a date flag could be wrong. Editorial verification would need separate evidence; this exercise tests the mechanical routing contract only.

Write ordinary and difficult cases

Each case has a unique identifier, input and expected outcome. Start with one ready example and one conflicting example, then add missing evidence, boundary values and ambiguous requirements. Keep the expected outcome tied to a written rule. A case where the candidate should stop can be as valuable as a case where it should complete. The evaluator rejects duplicate identifiers so a report cannot quietly collapse distinct examples under one convenient name. Bound the set so the local run remains easy to understand.

Run and inspect failures

Save local_evaluation.py and run it with Python 3.12. The main block evaluates the baseline against two fixed cases. test_workflows.py additionally introduces an always-draft candidate, a candidate that raises an exception and invalid input types. An exception becomes the observable outcome error without serializing its message. That avoids leaking the synthetic secret used by one test and illustrates why failure reporting should not automatically dump inputs or tool exceptions into a public scorecard.

Separate a harness check from a product decision

NIST’s framework supports thinking about evaluation within an intended context, but this fixture is not a NIST assessment or certification. A perfect score here means the local function matched these expected labels. It says nothing about research quality, autonomous tool behavior or an external provider. Before expanding the exercise, define which failures block use and which can remain visible in a supervised draft process. Avoid averaging an unauthorized action together with fluent prose until the serious failure disappears inside a composite score.

Extend the test without contaminating it

Keep a small holdout set that is not consulted while tuning a future prompt or rule. Record the candidate version and preserve changed expectations with a reason. If reviewers disagree, improve the task definition rather than silently changing the answer to favor one candidate. A later model adapter should have explicit data-handling and budget limits, and its actual network behavior needs its own tests. Keep the harness deterministic so an evaluator change cannot be confused with a candidate improvement.

Use the result to choose a next step

Inspect failed cases first and identify whether the problem lies in a missing definition, the candidate or the evaluator. Preserve a concise report that another reviewer can reproduce with the supplied files. A useful evaluation leaves the team knowing what was tested and what remains uncertain. That is a better foundation for a pilot than a large percentage whose inputs, acceptance rules and failure behavior cannot be inspected.

Runnable Python fixture

"""A deterministic fixture evaluator; no language model or network benchmark."""
def baseline(item):
    if (type(item) is not dict or set(item) != {"sources", "date_verified", "conflict"}
            or type(item["sources"]) is not int or item["sources"] < 0
            or type(item["date_verified"]) is not bool or type(item["conflict"]) is not bool):
        raise ValueError("invalid case input")
    return "draft" if item["sources"] >= 2 and item["date_verified"] and not item["conflict"] else "review"

def evaluate(candidate, cases):
    if type(cases) is not list or len(cases) > 100:
        raise ValueError("bounded case list required")
    seen, results = set(), []
    for case in cases:
        if type(case) is not dict or set(case) != {"id", "input", "expected"}:
            raise ValueError("invalid case shape")
        case_id = case["id"]
        if type(case_id) is not str or not case_id or len(case_id) > 60 or case_id in seen:
            raise ValueError("invalid or duplicate case id")
        if type(case["expected"]) is not str or case["expected"] not in {"draft", "review"}:
            raise ValueError("unknown expected outcome")
        seen.add(case_id)
        try:
            observed = candidate(case["input"])
        except Exception:
            observed = "error"  # Never serialize an exception containing input data.
        if observed not in ("draft", "review"):
            observed = "error"
        results.append({"id": case_id, "expected": case["expected"], "observed": observed,
                        "passed": observed == case["expected"]})
    return {"passed": sum(row["passed"] for row in results), "total": len(results), "cases": results}

if __name__ == "__main__":
    fixture = [{"id": "ready", "input": {"sources": 2, "date_verified": True, "conflict": False},
                "expected": "draft"},
               {"id": "conflict", "input": {"sources": 3, "date_verified": True, "conflict": True},
                "expected": "review"}]
    print(evaluate(baseline, fixture))

Verification and references

The evaluator passed five fixture tests on 16 October 2026 using Python 3.12.10, including deliberate candidate failures. These are harness tests, not an AI benchmark. Run local_evaluation.py for the example and test_workflows.py for the complete suite.

• Python 3.12 unittest documentation (living reference; accessed 2026-10-16)
• NIST AI Risk Management Framework (living reference; accessed 2026-10-16)