01 · Define
Set the unit of work, rubric, domain, languages, sensitive-data rules and acceptance thresholds.
Human AI evaluation · AI quality assurance
Add calibrated human judgment where AI systems need it.
Build a review operation around your rubric, domain, languages, security rules and volume, with named quality and adjudication responsibilities.
Send a sample task, rubric and expected volume.
The operating workflow
From sample task to controlled production review.
02 · Calibrate
Run reviewers against shared examples, inspect disagreement and refine instructions before volume.
03 · Evaluate
Assign work with traceable reviewer ownership, sampling, exception handling and progress reporting.
04 · Adjudicate
Route disagreements and edge cases to the named expert, then feed decisions back into calibration.
First-scope inputs
The details that let us shape a team.
- Sample tasks and current rubric
- Domain, language and reviewer requirements
- Expected volume, cadence and turnaround
- Data access, privacy and location constraints
- Quality thresholds and adjudication owner
What comes back
A plan you can check line by line.
- Reviewer capability and sourcing brief
- Calibration and disagreement workflow
- Quality sampling and adjudication plan
- Capacity assumptions and delivery cadence
- Responsibilities, dependencies and pricing details
Illustrative scope only. Reviewer availability, launch timing and pricing are confirmed after requirements and fit are validated.
Give the first evaluation workstream a trained team.
Send a sample task, rubric and expected volume. We respond with reviewer requirements, quality ownership and the inputs required to launch. A call is optional.
Give the first workstream a trained team.
Send the task, domain or language and approximate volume. No sensitive data. A person replies within 1 business hour with fit, missing inputs and the next step.
Common questions
What buyers ask before scoping evaluation work.
- Is this AI quality assurance or software testing?
- It is AI quality assurance done by people: trained reviewers check AI answers and outputs against your rubric. It is not automated software testing.
- Do you provide data annotation, AI trainers and RLHF review?
- Yes, when the work can be checked against written instructions. XCAILE scopes the task, sources and tests annotators and AI trainers, runs calibration and quality checks, and confirms real capacity before launch.
- What can people evaluate?
- Model outputs, conversations, documents, images, workflows, policy adherence, domain accuracy, language quality, and other reviewable work defined in the scope.
- How are reviewers calibrated?
- The scope defines a rubric, sample set, calibration round, disagreement rules, quality checks, and an adjudication route before production volume begins.
- Does XCAILE already have every reviewer available?
- No availability is assumed. XCAILE defines the capability brief, sources and validates relevant reviewers, and confirms actual capacity before proposing a launch plan.
- Who makes release or model decisions?
- Your team retains product, methodology, safety, and release decisions unless a written scope explicitly assigns a bounded responsibility elsewhere.
- How do we scale human review without lowering quality?
- Scale through the process, not by adding untrained volume: a defined rubric, calibration rounds, disagreement rules and quality checks set the standard, and reviewers join in waves against that standard. XCAILE scopes and manages the quality workflow. Send a sample task and your rubric.
- Can reviewers work inside our evaluation tooling?
- Yes, within the access you define. Reviewers work inside your tooling and annotate against your rubric; your team keeps methodology and release decisions. Send the tooling and access rules with the task.
- How do you staff rare domain or language expertise?
- XCAILE defines the capability brief, sources candidates with the domain or language profile, tests them against a sample of your tasks and confirms real capacity before launch. Nothing is assumed available. Send the domain, the languages and a sample task.
- How do you handle confidential model data and outputs?
- Documented security and data-handling controls, shared during procurement cover the handling of your outputs and data. Your team decides what reviewers can see and how access is scoped. Send the confidentiality requirements with the task.
- How is evaluation work priced?
- Pricing is sent in writing after you share the work: task type, volume, languages, quality bar and review cadence. Month to month, no minimum term.