Testing and Evaluating Agents

Use the right validation workflow before changing agents that support production decisions.

On this page

Choose a validation method

Parsewise provides several related but distinct checks:

  • Quality Check reviews whether an extraction task is well formed.
  • Golden set test runs a candidate configuration against a reference project and compares its results.
  • Evaluation agent scores a business criterion over other agents’ results.
  • Inconsistency detection identifies conflicting source extractions within a normal result.

Quality Check and golden-set testing help you improve an agent. Evaluation agents and inconsistency detection produce project results for users to review.

Run a Quality Check

Open an extraction agent and select Quality Check. The check focuses on two common sources of unreliable behavior:

  • Whether the task asks for one focused, page-level concept.
  • Whether it states a clear response shape for values that are not ordinary prose.

Apply suggestions that make the requested value or format less ambiguous. A Quality Check does not run the agent against documents and cannot prove that its results are correct; use a golden set for that.

Quality Check is not shown for derived agents because their prompts operate on resolved agent values rather than page-level document extraction.

Prepare a golden set

A golden set is a per-document validation project containing reviewed results. Its agent names act as the identity used to compare candidate and reference configurations.

Before testing:

  1. Create or choose a per-document validation project.
  2. Add representative documents, including difficult edge cases.
  3. Review and correct the reference results.
  4. Keep agent names aligned with the working project.

The selected validation project must contain the agents required by the working project. Parsewise remembers the choice for that project in the current browser.

Test against a golden set

From the agent editor, select Test against golden set. The test uses the configuration currently in the editor, including unsaved changes:

  1. Parsewise creates a temporary test agent in the validation project.
  2. It launches the candidate configuration against the golden documents.
  3. It compares the candidate with the same-named golden agent.
  4. The summary reports matching, different, missing, and candidate-only rows.

Select View full comparison to inspect each difference. If there is no same-named golden agent, the test can still show the candidate results, but it cannot calculate a comparison.

After review:

  • Promote to golden set replaces the golden agent with the tested configuration and its results.
  • Dismiss deletes the temporary test agent without changing the golden set.

Do not promote merely because the aggregate match rate is high. Inspect material mismatches and missing rows first.

Create an evaluation agent

Evaluation agents are derived agents that return integer scores for business criteria.

To create one:

  1. Create or edit a derived agent.
  2. Open Advanced and enable Evaluation agent.
  3. Reference at least one source agent.
  4. Write the criterion and scoring rules in the Derived Task.
  5. State what each score means and what evidence is required.
  6. Save and launch.

For example, an evaluation agent might return 1 when a contract’s termination right meets an approved standard and 0 when it does not. Use explicit, auditable rules rather than asking for a vague assessment.

Review evaluations

When the Evaluation Engine is enabled, open Evaluation in the Apps section. The page lists each evaluation criterion with its score and justification.

From there you can:

  • Launch pending evaluation agents.
  • Open a criterion to inspect the underlying result and sources.
  • Invert score coloring when lower or higher numbers represent the preferred outcome.
  • Download scores and justifications as Excel.

Evaluation agents run after the agents they reference. If an upstream result changes, the evaluation becomes pending and recomputes in dependency order.

Availability

Quality Check and golden-set controls appear only where their prerequisites are satisfied. Golden-set testing requires a suitable per-document validation project.

Evaluation agents and the Evaluation page require the organisation’s Evaluation Engine feature. Other roles or read-only users may not see creation and promotion controls. Contact support@parsewise.ai if your workflow requires evaluation access.