Exam CCAO-F Topic 1 Question 153 Discussion

Answer B is correct. The pilot has already demonstrated that the core approach is effective, so the team should retain the successful elements while making targeted changes for the two identified improvement opportunities. A controlled comparison before full rollout then provides evidence that the revised approach performs at least as well as the original pilot and that the modifications produce the expected improvement.
Option A ignores known improvement opportunities and unnecessarily carries identified weaknesses into production. Option C implements the changes during the broader rollout without validating them first.
Monitoring is necessary, but it is not a substitute for controlled pre-rollout testing because defects could affect many more users. Option D tests only the two opportunities in isolation. This may fail to evaluate how the new changes interact with the rest of the successful workflow.
A controlled comparison can evaluate the pilot configuration and the revised configuration against the same acceptance criteria, representative cases, and quality measures. This preserves what worked while reducing the risk of unintended regression. Anthropic's evaluation guidance recommends measurable success criteria, task-specific tests, and evaluations that distinguish between competing approaches. Therefore, B represents the strongest evidence-based adjustment before scaling the workflow. Anthropic: Define success criteria and build evaluations