AI evaluation and rollout
AI System Evaluation
For a team with a generative AI tool in pilot or production and no measured answer to how often it is wrong. A test set built with your reviewers, and a report on accuracy, citation and refusal behavior, with the logging and review steps the tool is missing.
- Price
- From $7,500
- Duration
- Two to four weeks
- Part of
- Document Intelligence
Who it is for
- Teams with a generative AI assistant or document tool in pilot or production, and no measured answer to how often it is wrong.
- Risk, compliance and audit teams asked to sign off on an AI tool.
- Vendors who need evidence of how their system behaves, for a buyer or a regulator.
What you receive
- Test set of real questions and documents, built with your staff, that you keep
- Results by error type, with examples
- Review of logging, human review and escalation
- Remediation plan, and model documentation written for a risk or compliance reader
An evaluation, not an audit, a certification or legal advice. Credited in full against a Build that starts within 60 days.
Two to four weeks, in steps
Week 1: the test set
Real questions and documents chosen with your reviewers, with the expected answer or behavior for each.
Week 2: the runs
The test set run against the system; results by error type, with examples; logging, human review and escalation reviewed.
Weeks 3 and 4: remediation
The remediation plan and model documentation written for a risk or compliance reader. A small system finishes in two weeks.
What we need from you
- Access to the system, through its interface or its API.
- A sample of the documents and the questions it handles.
- Two or three reviewers who know what a right answer is.
Scope, price and dates are confirmed in writing before the work starts. You own what is delivered. A complete example is published: the sample deliverable.
About the AI System Evaluation
Is this an audit or a certification?
No. It is an evaluation with its method stated. Audits and certifications stay with the firms licensed to issue them.
Can you evaluate a tool we bought rather than built?
Yes, through the interface or the API your team uses.
Can we rerun it after a change?
Yes. You keep the test set, with the expected answer or behavior for each question.
What does a result look like?
The sample deliverable shows a full evaluation of DF1's own chart assistant: the test set, results by error type, two defects found, and the remediation plan.
Other packages
AI Team Rollout (Claude or ChatGPT)
From $4,900, about two weeks. A working setup for a team of 5 to 50: permissions checked, up to five tools connected, one workflow built for the work the team does most, and a training session on the documents the team works with.
OpenReconciliation Automation Pilot
$9,500 fixed, three weeks. One reconciliation, automated end to end on your own exports: bank to ledger, subledger to general ledger, or intercompany.
OpenFinancial Model Review
$1,900 fixed, five business days. Your model read the way a lender, buyer or auditor will read it.
OpenFrom $7,500, two to four weeks. Scope confirmed in writing first.
Tell us your situation and when you need the result. A reply comes within one business day, from the person who would do the work.