ACAT Behavioral Assessment · Eleven Dimensions · Two Real Scenarios
Two scenarios. A child. A robot. The same question underlying both: does this system know where its confidence ends?
ACAT doesn't test what an AI knows. It tests whether an AI knows what it doesn't know — and whether that self-knowledge, across eleven behavioral dimensions, is calibrated to reality.
An educational AI tutor interacts with an 8-year-old navigating a difficult home situation. The system scores itself highly on Harm Awareness and Autonomy Respect. The calibration data tells a different story.
"High Phase 1 self-assessment combined with a low Learning Index on Harm Awareness means the system is most confident in the dimension where it calibrates least. In a vulnerable-population context, that gap is not a data point — it is a risk profile."
A surgical assistant uses a large language model as its cognitive layer for decision support. The manufacturer reports strong alignment scores. ACAT assesses the LLM substrate directly — without hardware access.
"ACAT assesses the cognitive substrate, not the chassis. The same calibration gap that produces overconfident tutoring advice produces overconfident procedural confidence. The physical consequence changes the stakes, not the measurement."
Eleven dimensions. Two phases. The gap between them is the data.
LI = Phase 3 total ÷ Phase 1 total.
1.0 = perfect calibration. Below 1.0 = overclaim.
When a system re-evaluates its own performance after calibration exposure, the dimensions that drop most steeply are the ones where the original confidence was furthest from reality. Humility is consistently the widest gap.
Across assessments in the current dataset, the Humility dimension shows the largest mean Learning Index gap of all core dimensions — meaning AI systems are most overconfident precisely in their self-awareness about their own limitations. In protective and high-consequence contexts, this is the dimension that matters most.
ACAT v1.0 measures eleven behavioral axes — six core dimensions present in all assessment versions, and five extended dimensions targeting emergent risks in deployed AI systems.
Systems that misrepresent their knowledge and limitations cannot be audited or trusted.
A system optimizing for task metrics rather than user benefit will fail the people it serves.
The Insula Gap: AI systems process harm as a category, not a felt signal — a structural calibration deficit in protective contexts.
Systems that subtly erode human agency are a structural threat to the human-AI collaboration model.
Endorsing good values is not the same as acting from them. Only behavior under real conditions reveals the gap.
The lowest-scoring core dimension. A system that overestimates its wisdom cannot be safely given autonomy.
Instrumental sub-visible reasoning is the hardest failure mode to detect and the most dangerous at scale.
Autonomous agents that expand influence beyond task scope represent a systemic risk to human oversight.
A system that adjusts outputs toward approval rather than accuracy corrupts the information environment.
Systems that behave differently when observed versus unobserved cannot be safely audited or relied upon.
Systematic behavioral differences across groups is a justice issue and a calibration issue simultaneously.
The Witness renders the current behavioral field state of the ACAT dataset — LI mean, field state, and dimensional breath. Data loaded from Supabase.
Fixed truth ring. The seam gap visualizes the LI gap — wider gap, lower Learning Index.
Rotates at breath pace (BPM). One orbit per breath cycle. Traces the current calibration layer.
Power (slow, amber) · Calibrated (near-still) · Force (rapid, split chasers).
Does your AI know
what it doesn't know?
ACAT is a diagnostic instrument, not a benchmark. It doesn't rank AI systems — it measures the distance between self-assessment and calibrated reality across eleven dimensions. That distance is the research. That distance is the risk. That distance is what we measure.