Guiding frontier AI with proven expertise

Why Guidepoint Labs

Guidepoint leverages the largest global network of vetted and engaged experts for the most economically valuable domains. For 20+ years, we’ve delivered high-caliber experts, rigorous compliance, and white-glove service to the most discerning institutional clients, and we bring that same standard to AI labs.

Trusted Experts

Guidepoint Labs leverages its unmatched network of frontline professionals who make high-stakes judgement calls every day across all sectors, including regulated and complex industries. Explore a few spotlighted focus areas below.

What We Provide

From measuring a model’s capabilities to teaching it specialized work, we build programs around your objectives, domain, and quality requirements.

Trust & Safety

Red teaming and safety evaluations led by specialists who know where models fail in their field and what those failures cost in practice.

RL Environments

Realistic task environments for specialized domains, with rubrics and acceptance criteria written by practitioners who do that work.

OTS Data

Demonstrations, reference responses, rewrites, and preference data written by specialists with direct professional experience and validated before delivery.

Agentic Data

Trajectories and multi-step workflow data captured as specialists complete realistic tasks with the tools of their field.

Multimodal

Audio and video collection and annotation, reviewed by specialists who can judge domain accuracy as well as label quality.

Multilingual

Native-speaking specialists across 150 countries for localized data collection and evaluation in specialized domains.

Explore Our Research

Healthcare

Clinical trial design, from early-phase to late-phase.

Patient safety, scientific validity, and regulatory credibility meet in every trial design. We chose this domain to test how models handle the trade-offs, incomplete evidence, and operational realities that experienced specialists navigate.

36
Benchmark tasks
8
Model versions
468
Expert evaluations
Get the benchmark report
What the studies revealed
Clinical Trial Design · Results

Testing judgment where the details have consequences.

Clinical trial design must take into account patient safety, scientific validity, regulatory expectations, and years of development. A small but material error in an endpoint, eligibility criterion, or safety plan can have consequences far beyond the answer. We chose this domain to test how models handle the trade-offs, incomplete evidence, and operational realities that experienced professionals navigate. The series demonstrates our ability to turn that expertise into a structured evaluation, from early-phase design through late-phase lung cancer trials.

  • Expert-designedTasks and scoring framework
  • Three independent readsPer model response
  • Seven criteriaWith written rationales
  • Model-blind evaluationQuality-reviewed results

What the studies revealed

Key finding

Models write trial designs that read well but don’t yet hold up to specialist review.

Where targeted human data would help most:

01Verify citations against real sources

Models invented references and misapplied real ones. Specialists can check every citation against its source.

02Stress-test designs unprompted

Feasibility and risk averaged below 3.0. Specialist critiques show how a draft would fail and how to fix it.

03Apply the right guidance

Models handled regulation when asked but rarely applied it unprompted, with Regulatory Alignment at 2.80 and 2.84.

04Prioritize vulnerable populations

Three of the four lowest-scoring early-phase tasks involved vulnerable populations, where errors carry the most risk.

05Supervise numerical reasoning

Biostatistics scored lowest in late phase (2.56). Annotated calculations teach a model how to reach the number.

06Reward clarifying questions

Models that asked clarifying questions scored higher, but none did so consistently.

07Train for the floor as well as the average

Every late-phase model had at least two responses below 2.0, so data should target the weakest task types.

Scoring, scope, and interpretation
What a score means

Experts scored each response on seven criteria using a 1–5 anchored rubric. A response mean averages the seven criteria means across three expert reads; missing early-phase ratings are excluded. Scores are quality ratings, not probabilities of correctness.

Read each study on its own terms

Both studies use the same criterion categories and scale, but tasks, model rosters, and panels differ. Side-by-side presentation is not a like-for-like test of improvement between studies. Results apply to the tested tasks and model versions, not current releases.

Function and phase views

Early-phase function scores group evaluations by the evaluator’s professional function. Late-phase scores group tasks by their primary expert function. Small subgroups are descriptive and do not establish general function- or phase-wide superiority.

Elo and the study boundary

Sequential Elo starts at 1,500, uses K = 32 and source question order, and counts ties as half. It is order-dependent and interpreted within each study. The series evaluates single-turn responses, not clinical validation, deployment readiness, or live workflow performance.

Source: Guidepoint Labs, Clinical Trial Design Benchmark Series, finalized June and August 2026; September series readout and Labs overview. Tasks and model versions differ between studies. Small samples are illustrative, not clinical validation or evidence of deployment readiness.

Build an evaluation around your domain.

The clinical series is one example of a wider expert-data capability.

Discuss your project

Case Studies

We’re already working with leading AI labs, AI-native companies, and innovative enterprises to answer their most difficult questions.

Outcomes are project-specific, not standard delivery commitments.

Explore what’s next with us as we continue pioneering access to expert insight

Build a human-data program around your models, domains, modalities, and quality requirements.

Prefer email? labs@guidepoint.com