Exam CCAR-F Topic 1 Question 53 Discussion

An aggregate accuracy value can conceal severe performance disparities. A system may achieve 97% overall accuracy while performing poorly on a low-volume document type, a critical financial field, or a specific edge case. Automating outputs solely from the aggregate figure could therefore expose downstream systems to concentrated, high-impact errors.
Anthropic's evaluation guidance states that evaluations should be task-specific, reflect the real-world task distribution, and explicitly include edge cases. It also emphasizes multidimensional success criteria rather than reliance on a single global metric. ( https://docs.anthropic.com/en/docs/build-with-claude/develop-tests ) Option A applies those principles by stratifying performance according to document type and field. This reveals whether confidence is calibrated consistently and whether the proposed automation threshold remains safe for every operationally significant segment.
Option B is useful only after segment-level performance has been understood. Selecting a global threshold cannot correct a subgroup where confidence is systematically overstated. Option C is necessary governance work, but it treats the overall 97% result as though errors were uniformly distributed. Option D places unvalidated outputs into downstream systems and depends on passive error reporting, which may fail to detect silent corruption.
The correct deployment gate is therefore segmented validation, followed by threshold selection, downstream acceptance criteria, and a controlled pilot.
Official references/topics: Define Success Criteria; Task-Specific Evaluations; Edge-Case Coverage; Reliability Segmentation.