Share GPUs on Amazon ECS

Many GPU workloads, such as inference and small model serving, use only a fraction of a GPU's compute and memory. Without sharing, the rest of that capacity sits idle. With Amazon ECS, you can share a single physical NVIDIA GPU across multiple containers so that you can pack more workloads onto the same hardware and reduce GPU waste.

With Amazon ECS, you can use NVIDIA Multi-Process Service (MPS) for GPU sharing. MPS is an NVIDIA CUDA feature that runs multiple processes concurrently on the same GPU, which improves GPU utilization and reduces context-switching overhead. MPS provides memory and compute partitioning, so each container gets a configurable memory limit and an optional compute ceiling.

Amazon ECS manages MPS for you. It starts the MPS daemon on GPU instances, places containers onto GPUs based on available memory, and enforces per-container limits.

MPS provides software-level sharing rather than hardware-level partitioning. For more information about MPS, see Multi-Process Service in the NVIDIA documentation.

Amazon ECS supports GPU sharing with MPS on Amazon ECS Managed Instances and Amazon ECS on Amazon EC2.

When to use GPU sharing

GPU sharing with MPS is well suited for:

GPU sharing is not recommended for:

How GPU sharing works

You configure GPU sharing in the container definition of a task definition. In each container's resourceRequirements, you provide a sharingStrategy with a GPU memory value instead of specifying an integer in value for a dedicated GPU.

When Amazon ECS places a task that uses GPU sharing, it adds up the GPU memory requested by all GPU containers in the task and looks for a GPU with enough free memory. All GPU containers within a single task are placed on the same physical GPU. If the task requests 4,000 MiB across two containers, Amazon ECS finds a single GPU with at least 4,000 MiB of remaining memory and places both containers there.

Multiple tasks can also share the same physical GPU. As long as a GPU has enough remaining memory, Amazon ECS can place additional tasks on it, even if those tasks come from different services. A task that shares a GPU and a task that uses a dedicated GPU can run on the same instance, on different GPUs.

On Amazon ECS Managed Instances, Amazon ECS monitors the health of GPUs and the MPS daemon on each instance. If Amazon ECS detects a GPU hardware failure or an MPS daemon failure, it drains the instance and replaces it. For more information about GPU auto repair, see GPU auto repair for Amazon ECS managed instances.

Configure GPU sharing

To share a GPU, add a sharingStrategy object to the resourceRequirements of each GPU container in your task definition. The sharingStrategy object contains an nvidiaMps object with the following fields. For more information about resourceRequirements, see Resource requirements.

Amazon ECS places tasks based on GPU memory only, so it doesn't account for compute when making placement decisions.

You can set the sharing strategy in the Amazon ECS console task-definition editor, or with JSON, the AWS CLI, or the API.

Amazon ECS sets the following reserved environment variables in each container that shares a GPU. Don't set these variables in your container definition.

The following example task definition runs two inference containers that share one GPU with MPS, along with one non-GPU sidecar container. Each GPU container requests a different amount of GPU memory and compute.

In the following example, replace account-id with your AWS account ID.

{
  "family": "mps-inference",
  "networkMode": "awsvpc",
  "executionRoleArn": "arn:aws:iam::account-id:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "name": "model-a",
      "image": "nvidia/cuda:12.0.0-base-ubuntu22.04",
      "essential": true,
      "resourceRequirements": [
        {
          "type": "GPU",
          "sharingStrategy": {
            "nvidiaMps": {
              "memory": 12288,
              "maxComputePercent": 50
            }
          }
        }
      ]
    },
    {
      "name": "model-b",
      "image": "nvidia/cuda:12.0.0-base-ubuntu22.04",
      "essential": true,
      "resourceRequirements": [
        {
          "type": "GPU",
          "sharingStrategy": {
            "nvidiaMps": {
              "memory": 8192,
              "maxComputePercent": 25
            }
          }
        }
      ]
    },
    {
      "name": "log-router",
      "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:stable",
      "essential": false
    }
  ],
  "requiresCompatibilities": [
    "MANAGED_INSTANCES"
  ],
  "cpu": "4096",
  "memory": "8192"
}

Validation rules

Amazon ECS applies the following rules to GPU sharing configurations:

Set up GPU sharing by launch type

With Amazon ECS Managed Instances, Amazon ECS selects and manages the GPU instances for you. You don't need any setup beyond the task definition.

With Amazon ECS on Amazon EC2, launch a supported NVIDIA GPU instance with the Amazon ECS AL2023 GPU-optimized AMI. GPU sharing with MPS on Amazon ECS on Amazon EC2 requires Amazon ECS container agent version 1.107.0 or later and the Amazon ECS AL2023 GPU-optimized AMI version 20261019 or later. After the instance registers with your cluster, the task-definition configuration is the same as for Amazon ECS Managed Instances.

Considerations

Monitor GPU sharing

You can confirm that a GPU is shared and track GPU usage with the Amazon ECS API and Amazon CloudWatch metrics.

Confirm a GPU is shared

To confirm that a GPU is shared, call the DescribeContainerInstances API. The response includes the gpuDevices field with the sharing mode, container count, and remaining memory for each GPU. The following is an example response.

{
  "containerInstances": [
    {
      "ec2InstanceId": "i-0c74b75400ab4228d",
      "runningTasksCount": 1,
      "gpuDevices": [
        {
          "gpuId": "GPU-70d3bdaf-8635-81e0-8e44-446248a92c83",
          "sharingMode": "NVIDIA_MPS",
          "totalMemoryInMiB": 22587,
          "remainingMemoryInMiB": 16587,
          "containerCount": 2
        }
      ]
    }
  ]
}

View per-task configuration

To view the sharing configuration for each task, call the DescribeTasks API. Each container's gpuSharingStrategy shows its memory and compute limits, and containers that share a GPU list the same value in gpuIds. The following is an example response.

{
  "tasks": [
    {
      "lastStatus": "RUNNING",
      "launchType": "MANAGED_INSTANCES",
      "containers": [
        {
          "name": "mps-a",
          "gpuIds": ["GPU-70d3bdaf-8635-81e0-8e44-446248a92c83"],
          "gpuSharingStrategy": { "nvidiaMps": { "memory": 4000, "maxComputePercent": 50 } }
        },
        {
          "name": "mps-b",
          "gpuIds": ["GPU-70d3bdaf-8635-81e0-8e44-446248a92c83"],
          "gpuSharingStrategy": { "nvidiaMps": { "memory": 2000, "maxComputePercent": 50 } }
        }
      ]
    }
  ]
}

Track GPU usage