Amazon SageMaker AI inference now supports GPU partitioning for real-time endpoints

Amazon SageMaker AI Inference

Amazon SageMaker AI inference now supports GPU partitioning, enabling you to run several models on a single GPU instead of dedicating a whole accelerator to each one. Using NVIDIA Multi-Instance GPU (MIG) technology, SageMaker AI divides a GPU into as many as seven isolated partitions, each with its own memory and compute. This feature allows customers serving small and mid-size models to share a single accelerator, optimizing resource usage and cost.

What to do

• Configure GPU partitioning when creating an endpoint for inference components.
• Continue to invoke each model by name, as your application code remains unchanged.

• Item: Limit: GPU partitioning is available on P4d, P5, G7, and G7e instances.
• Regions: Available in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt), and Europe (Ireland).

Source: AWS release notes

If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.