AI evaluation and rollout

AI System Evaluation

For a team with a generative AI tool in pilot or production and no measured answer to how often it is wrong. A test set built with your reviewers, and a report on accuracy, citation and refusal behavior, with the logging and review steps the tool is missing.

Price
From $7,500
Duration
Two to four weeks
Part of
Document Intelligence
  • Teams with a generative AI assistant or document tool in pilot or production, and no measured answer to how often it is wrong.
  • Risk, compliance and audit teams asked to sign off on an AI tool.
  • Vendors who need evidence of how their system behaves, for a buyer or a regulator.
  • Test set of real questions and documents, built with your staff, that you keep
  • Results by error type, with examples
  • Review of logging, human review and escalation
  • Remediation plan, and model documentation written for a risk or compliance reader

An evaluation, not an audit, a certification or legal advice. Credited in full against a Build that starts within 60 days.

Two to four weeks, in steps

  1. Week 1: the test set

    Real questions and documents chosen with your reviewers, with the expected answer or behavior for each.

  2. Week 2: the runs

    The test set run against the system; results by error type, with examples; logging, human review and escalation reviewed.

  3. Weeks 3 and 4: remediation

    The remediation plan and model documentation written for a risk or compliance reader. A small system finishes in two weeks.

  • Access to the system, through its interface or its API.
  • A sample of the documents and the questions it handles.
  • Two or three reviewers who know what a right answer is.

Scope, price and dates are confirmed in writing before the work starts. You own what is delivered. A complete example is published: the sample deliverable.

Ask about this package

About the AI System Evaluation

Is this an audit or a certification?

No. It is an evaluation with its method stated. Audits and certifications stay with the firms licensed to issue them.

Can you evaluate a tool we bought rather than built?

Yes, through the interface or the API your team uses.

Can we rerun it after a change?

Yes. You keep the test set, with the expected answer or behavior for each question.

What does a result look like?

The sample deliverable shows a full evaluation of DF1's own chart assistant: the test set, results by error type, two defects found, and the remediation plan.

From $7,500, two to four weeks. Scope confirmed in writing first.

Tell us your situation and when you need the result. A reply comes within one business day, from the person who would do the work.