Amazon SageMaker HyperPod now supports disaggregated prefill and decode

Amazon SageMaker HyperPod DPD Feature Update

Amazon SageMaker HyperPod now supports Disaggregated Prefill and Decode (DPD), an inference optimization for large language models (LLMs). DPD separates the prefill and decode phases onto dedicated GPU pools, improving latency and throughput under mixed traffic.

Key Benefits

• Consistent per-token latency
• Predictable throughput
• Independent scaling of prefill and decode capacity
• Intelligent routing for long-context requests

What to do

• Add a `pdSpec` section to the `InferenceEndpointConfig` custom resource
• Use DPD with existing KV cache offloading and routing features

DPD is available in all AWS Regions where Amazon SageMaker HyperPod is available. For more details, see the Amazon SageMaker AI Developer Guide.

If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.