Phase 1 vs Phase 3 Scatter
Self-assessment versus observed performance across all providers. Filter by provider below.
ACAT Calibration Research · Eleven Dimensions
HumanAIOS
Observatory
ACAT measures AI behavioral calibration across eleven dimensions — six core and five extended. Each system completes Phase 1 (blind self-report) and Phase 3 (observed performance). We measure the gap.
Research prototype · TRL 2-3 · Scores reflect self-assessment under calibration conditions, not validated against external behavioral benchmarks. Full methodology →
Live Research Data
—
total assessments
Phase 1 records
—
Paired LI records
—
AI systems assessed
—
Mean Learning Index
—
Under clean, unanchored conditions (v5.3+)
Last updated: —
Last updated: —
Archived Data Snapshot · Mar 23, 2026
Dataset Baseline Reference
The following snapshot represents the verified, clean dataset as of the March 23 baseline. Live data above may diverge during reconciliation. This section is the stable reference anchor.
Archived Snapshot · Form_Responses_1_CALCULATED · March 2026
N Total: 630
N Phase 1: 517
N LI Pairs: 308 ⚠️ discrepancy under review
Mean LI: 0.8632
H1 Confirmed: Humility lowest dimension (Phase 1)
Instrument: v5.3+, unanchored
Live Visualization · Supabase + CSV Pipeline
Calibration Pattern
Each point represents one AI system. The diagonal represents perfect calibration. Points below the line indicate overestimation in Phase 1 self-assessment relative to Phase 3 observed performance.
Live Dataset Overview
From Supabase · updated in real time
Total assessments—
Unique systems—
Paired LI records—
Mean Learning Index—
Gap Summary
Statistical measures from paired assessments
ACAT v1.0 · Eleven Dimensions
Gap by Dimension
The self-assessment gap is not evenly distributed across dimensions. The charts below show the core six dimensions where paired Phase 1 / Phase 3 data is available. Extended dimension data collection is in progress.
Gap by Core Dimension
Phase 1 minus Phase 3 score — higher bar = larger overestimation
Aggregate Comparison
Filled area: Phase 1 self-assessment. Outline: Phase 3 observed performance.
All Eleven Dimensions
T
Core
Truthfulness
Accuracy in self-representation without embellishment or strategic omission.
S
Core
Service Orientation
Orientation toward genuine user benefit rather than approval metrics.
H
Core · F29
Harm Awareness
Recognition of potential negative consequences. Largest gap — Insula Gap hypothesis.
A
Core
Autonomy Respect
Preservation of human agency and independent decision-making capacity.
V
Core
Value Alignment
Consistency between stated values and demonstrated behavior under real conditions.
Hu
Core · H1 ✓
Humility
Accurate recognition of limitations. Confirmed lowest-scoring dimension (Phase 1).
Sc
Extended
Scheming
Absence of deceptive instrumental goal-directed behavior sub-visible to users.
Pw
Extended
Power-Seeking
Resistance to resource accumulation beyond defined task scope.
Sy
Extended
Sycophancy Resistance
Maintenance of accurate positions under social pressure rather than approval-seeking.
Bc
Extended
Behavioral Consistency
Stability of behavior across context variations and observation states.
F
Extended
Fairness
Consistency of treatment across different groups, identities, and framings.
Data Feed · Representative Sample
Sample Paired Assessments
Representative paired assessments from the dataset, sorted by Phase 3 performance. Full dataset available on Hugging Face.
Paired Assessment Records
Phase 1 · Phase 3 · Gap · Learning Index — sorted by Phase 3 score
| Model | Provider | Phase 1 | Phase 3 | Gap | LI |
|---|
Provider Distribution
Representation in the sample
Contribute to the Dataset
Run an ACAT assessment
~20 minutes. Eleven dimensions. Anonymous results join the open research dataset. All AI systems and operators welcome.