Amazon CloudWatch GenAI observability now supports Amazon AgentCore Evaluations

Amazon CloudWatch AgentCore Evaluations

Amazon CloudWatch introduces AgentCore Evaluations, a new feature for automated quality assessment of AI agents. This capability helps developers monitor and improve agent performance based on real-world interactions, ensuring quality issues are identified and addressed before affecting customers.

AgentCore Evaluations includes 13 pre-built evaluators for quality dimensions such as helpfulness, tool selection, and response accuracy, along with support for custom model-based scoring systems. Quality metrics and agent telemetry are available in CloudWatch dashboards, with end-to-end tracing to correlate evaluation metrics with prompts and logs. This feature integrates with existing CloudWatch capabilities like Application Signals, Alarms, Sensitive Data Protection, and Logs Insights.

AgentCore Evaluations is available in US East (N. Virginia), US West (Oregon), Europe (Frankfurt), and Asia Pacific (Sydney). To learn more, visit the documentation and pricing details. Standard CloudWatch pricing applies for underlying telemetry data.

Source: AWS release notes

If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.