Creating a compute node group in AWS PCS
This topic describes how to create a compute node group in AWS Parallel Computing Service (AWS PCS).
When you create a compute node group, you choose how to configure its compute nodes. With
standard configuration, you specify the AMI, security groups, and SSH key pair on the compute
node group. These are called instance settings. With launch template
configuration, you also use a custom Amazon EC2 launch template, for Amazon EC2 settings that aren't
available as instance settings, such as network interfaces or capacity reservations.
You can also configure custom Slurm settings, and node lifecycle actions that
run scripts at defined points in a compute node's lifecycle, such as to mount file systems or
install software. For more information, see Configuring custom Slurm settings in AWS PCS and Node lifecycle actions.
If this is your first time creating a compute node group in AWS PCS, we recommend that
you follow the tutorial in Get started with AWS Parallel Computing Service.
Prerequisites
-
Sufficient service quotas to launch the desired number of EC2 instances in your AWS Region.
You can use the Service Quotas console to check and request
increases to your service quotas.
-
An existing VPC and subnets that meet AWS PCS networking requirements.
We recommend that you understand these requirements before you deploy a
cluster for production use. For more information, see
AWS PCS VPC and subnet requirements and considerations.
You can also use an AWS CloudFormation template to create a VPC and subnets. AWS provides
an HPC recipe for the CloudFormation template. For more information, see
aws-hpc-recipes on GitHub.
-
One or more security groups in your cluster VPC that allow traffic between compute nodes
and the cluster. For more information, see Security group requirements and considerations.
-
An AMI that is compatible with AWS PCS. For production workloads, you can use the
AWS-maintained PCS-ready AWS Deep Learning AMI (DLAMI). If you're evaluating AWS PCS,
you can use a sample AMI.
You can also build your own AMI. For more information, see Amazon Machine Images (AMIs) for AWS PCS.
-
An IAM instance profile with permissions to call the AWS PCS
RegisterComputeNodeGroupInstance API action and access to any other AWS
resources that your compute nodes need. For more information,
see IAM instance profiles for AWS Parallel Computing Service.
-
Permissions for the IAM identity that creates the compute node group. If you specify an
SSH key pair, the identity also needs ec2:DescribeKeyPairs. For more information,
see Minimum permissions to use API actions.
-
The AWSServiceRoleForEC2Spot service-linked role in your AWS account, if
you create a compute node group that uses Spot Instances. For more information, see Amazon EC2 Spot role for AWS PCS.
-
A custom launch template, only if you need Amazon EC2 settings that aren't available as
instance settings, such as network interfaces or capacity reservations. A launch template can also provide the
AMI, security groups, and SSH key pair. For more information, see Using Amazon EC2 launch templates with AWS PCS and Configuring instance settings for compute node groups in AWS PCS.
How AWS PCS chooses the AMI, security groups, and SSH key pair
If you don't use a custom launch template, AWS PCS uses the AMI, security groups, and
SSH key pair that you specify on the compute node group.
If you use a custom launch template, you can specify each of these settings on the compute
node group, in the launch template, or in both places. AWS PCS chooses each setting
separately, as the following table shows. AWS PCS doesn't change your launch template.
Instance settings with a custom launch template
| Setting |
Specified on the compute node group |
Specified only in the launch template |
Not specified in either place |
AMI ID |
AWS PCS uses the AMI on the compute node group instead of any AMI in the
launch template. |
AWS PCS uses the AMI in the launch template. |
AWS PCS rejects the request. |
Security groups |
AWS PCS uses only the security groups on the compute node group. It doesn't
add the security groups in the launch template. |
AWS PCS uses the security groups in the launch template. |
AWS PCS rejects the request. |
SSH key pair |
AWS PCS uses the key pair on the compute node group instead of any key pair
in the launch template. |
AWS PCS uses the key pair in the launch template. |
Compute nodes don't have a key pair. |
For example, suppose that your launch template specifies an AMI and security group
sg-ExampleID1, and you specify only security group sg-ExampleID2 on
the compute node group. Compute nodes use the AMI from the launch template and only
sg-ExampleID2.
You can't specify security groups on the compute node group if your launch template defines
network interfaces. Specify the security groups for each network interface in the launch template
instead. For more information, see Multiple network interfaces in AWS PCS.
To change or remove these settings after you create the compute node group, see Configuring instance settings for compute node groups in AWS PCS.
Launch templates that AWS PCS creates
For each compute node group, AWS PCS creates a launch template of its own and uses it to
launch compute nodes. AWS PCS creates this launch template whether or not you use a custom
launch template. AWS PCS builds this launch template from the settings on the compute node
group and, if you use one, your custom launch template. AWS PCS doesn't change your launch
template.
The launch templates that AWS PCS creates are named
pcs-identifier-do-not-delete. They appear with your own
launch templates, for example, in the Amazon EC2 console.
Don't modify or delete a pcs-identifier-do-not-delete
launch template, and don't choose one as the custom launch template for a compute node group.
If you do, the compute node group won't work correctly.
Create a compute node group in AWS PCS
You can create a compute node group by using the AWS Management Console or the AWS CLI.
- AWS Management Console
-
To create a compute node group (console)
-
Open the AWS PCS console.
-
Choose the cluster where you want to create a compute node group. Choose
Compute node groups, and then choose Create.
-
Choose how to configure the compute nodes in the compute node group.
Standard configuration is selected by default.
-
Standard configuration – Configure compute nodes without
a launch template. You must provide an AMI ID and at least one security group. Compute
nodes require Instance Metadata Service Version 2 (IMDSv2).
-
Launch template configuration – Use a custom Amazon EC2 launch
template. A launch template can configure settings that aren't available as instance
settings. Examples include network interfaces for EFA, capacity reservations,
placement groups, block device mappings, and instance metadata options.
For more information, see Configuring instance settings for compute node groups in AWS PCS.
-
In the Compute node group setup section, enter a name for your
compute node group. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It
must start with an alphabetic character and be 3–25 characters long. The name
must be unique within the cluster.
-
Under Computing configuration, enter or choose these
values:
-
EC2 launch template – If you chose Launch
template configuration, choose a custom launch template to use for this node
group. To learn how to create a launch template, see Using Amazon EC2 launch templates with AWS PCS.
Don't choose a launch template named
pcs-identifier-do-not-delete. AWS PCS creates
launch templates with this name for its own use. For more information, see Launch templates that AWS PCS creates.
-
EC2 launch template version – If you chose
Launch template configuration, choose a version of your custom launch
template. If you change the version later, you must update the compute node group to
detect changes in the launch template. For more information, see
Updating a compute node group in AWS PCS.
-
AMI ID – Choose where to get the AMI, and then choose the
AMI:
-
PCS-ready AMIs – AWS-maintained AMIs for production
AI/ML and HPC workloads. This option is selected by default. For more information, see
Using PCS-ready DLAMI with AWS PCS.
-
PCS sample AMIs – Sample AMIs that AWS PCS provides,
for evaluation only. For more information, see Using sample Amazon Machine Images (AMIs) with AWS PCS.
-
Shared AMIs – AMIs that other AWS accounts share with you.
-
Private AMIs – AMIs that you created.
-
Custom AMI – Enter the ID of any AMI, such as an AMI from a
third-party vendor or from AWS Marketplace.
A shared, private, or custom AMI must be compatible with AWS PCS. For more
information, see Custom Amazon Machine Images (AMIs) for AWS PCS.
Whether you must choose an AMI depends on how you configure the compute nodes:
-
Standard configuration – Required.
-
Launch template configuration – Optional. If you
provide an AMI ID, AWS PCS uses it instead of the AMI in the launch template. If
you leave this field empty, AWS PCS uses the AMI in the launch template. If
neither the compute node group nor the launch template specifies an AMI, AWS PCS
rejects the request.
-
IAM instance profile – Choose an instance profile for
the compute node group. An instance profile grants compute nodes permissions to access AWS
resources and services. If you don't have an instance profile, choose Create a
basic profile. AWS PCS creates one with the minimum policy. To create your
own, see IAM instance profiles for AWS Parallel Computing Service.
-
Instances – Choose one or more instance types to fulfill
scaling requests in the node group. All instance types must have the same processor
architecture (x86_64 or arm64) and number of vCPUs. If the instance types have
GPUs, all instance types must have the same number of GPUs.
-
Scaling configuration – Specify the minimum and maximum
number of instances for the node group. Set the minimum (min) equal to the maximum (max)
for static capacity (for example, 5 min, 5 max). Set the minimum to 0 for fully dynamic
scaling (for example, 0 min, 10 max). With Slurm 24.05 or later, you can
set the minimum greater than 0 and less than the maximum for mixed capacity. This
maintains a baseline number of instances and scales up as needed (for example, 2 min,
10 max).
-
Under Network, enter or choose these values:
-
Subnets – Choose one or more subnets in the VPC where
your AWS PCS cluster is deployed. If you choose multiple subnets, EFA communications
won't be available between nodes, and communication between nodes in different subnets
might have increased latency. If you use a launch template, make sure that the subnets
you specify here match any subnets in the launch template. For On-Demand node groups
with more than one subnet, the order in which you list the subnets sets the launch
priority.
-
Security groups – Choose one or more security groups in
the VPC where your AWS PCS cluster is deployed. They must allow traffic between compute
nodes and the cluster. For more information, see Security group requirements and considerations.
-
Standard configuration – Required. Choose at least one
security group.
-
Launch template configuration – Optional. If you choose
security groups, AWS PCS uses only those security groups. It doesn't add the
security groups in the launch template. If you don't choose any, AWS PCS uses the
security groups in the launch template. If neither the compute node group nor the launch
template specifies security groups, AWS PCS rejects the request. You can't choose security groups if your launch template defines
network interfaces.
-
SSH key pair – Optional. Choose an Amazon EC2 key pair for SSH
connections to compute nodes.
-
Standard configuration – Choose a key pair, or choose
not to use one. If you don't use a key pair, compute nodes don't have one.
-
Launch template configuration – If you choose a key
pair, AWS PCS uses it instead of the key pair in the launch template. If you
choose to use the key pair in the launch template, AWS PCS uses that key pair. If
neither the compute node group nor the launch template specifies a key pair, compute
nodes don't have one.
For a summary of how AWS PCS chooses each setting, see How AWS PCS chooses the AMI, security groups, and SSH key pair.
-
(Optional) Under Additional settings, specify the
following:
-
Purchase option – Choose On-Demand Instances,
Spot Instances, an Interruptible Capacity Reservation, or an existing Capacity Block.
Choose On-Demand if you plan to use an On-Demand Capacity Reservation
(ODCR). For more information, see Using ODCRs with AWS PCS. Choose Interruptible Capacity
Reservation to use a shared interruptible ODCR (I-ODCR). For more information,
see Using I-ODCRs with AWS PCS. Choose Capacity Block
to use an existing Amazon EC2 Capacity Blocks for ML reservation. For more information, see
Using Amazon EC2 Capacity Blocks for ML with AWS PCS.
To use an ODCR, an I-ODCR, or a Capacity Block, you must choose Launch
template configuration and use a launch template that targets the
reservation.
-
Allocation strategy – If you chose the Spot
purchase option, you can specify how AWS PCS chooses Spot capacity pools when it
launches instances in the compute node group. For more information, see Allocation strategies for Spot Instances in the Amazon Elastic Compute Cloud User Guide.
This option has no effect if you chose the On-Demand purchase option.
-
(Optional) In the Scheduler configuration section, you can
specify the following:
-
Scale-down idle time – The time in seconds before
AWS PCS scales down an idle node in this compute node group. If you don't set it,
AWS PCS uses the cluster-level value. This setting requires Slurm version 25.11 or later.
-
(Optional) In the Slurm custom settings section,
you can add parameter name and value pairs to configure additional Slurm settings. For a complete list of supported parameters, see Custom Slurm settings for AWS PCS compute node groups.
-
(Optional) In the Node lifecycle actions section, you can add
scripts that run at defined points in a compute node's lifecycle. Choose Add
script. Choose the lifecycle stage, and specify the script location, a name,
optional arguments, error handling behavior, and execution policy. To add more scripts,
repeat these steps. Scripts run from top to bottom within a stage. To change the execution
order, choose Actions for a script, and then choose Move
up or Move down. For more information,
see Node lifecycle actions.
-
(Optional) Under Tags, add any tags to your compute node
group.
-
Choose Create compute node group. The Status
field shows Creating while AWS PCS provisions the compute node group.
Provisioning can take several minutes.
- AWS CLI
-
To create a compute node group (AWS CLI)
The following command creates a compute node group without a custom launch template. It
sets --launch-template-mode to MANAGED, which is the standard
configuration in the console. You must specify MANAGED because the default mode
is CUSTOM, which requires a custom launch template. To use a custom launch
template, see the example that follows the procedure. Before you run the command, make the
following replacements:
-
Replace region with the ID of the AWS Region where your
cluster is, such as us-east-1.
-
Replace my-cluster with the name or clusterId of
your cluster.
-
Replace my-node-group with the name for your compute node
group. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It
must start with an alphabetic character and be 3–25 characters long. The name
must be unique within the cluster.
-
Replace ami-ExampleID1 with the ID of an AMI that is
compatible with AWS PCS, such as the PCS-ready DLAMI. To get the ID of the current
PCS-ready DLAMI, see Find the current PCS-ready DLAMI. For other AMI options, see Amazon Machine Images (AMIs) for AWS PCS.
-
Replace subnet-ExampleID1 with one or more subnet IDs from
your cluster VPC. Separate multiple IDs with spaces. If you specify multiple subnets, EFA
communications won't be available between nodes. For On-Demand node groups with more than
one subnet, the order of the subnets sets the launch priority.
-
Replace sg-ExampleID1 with one or more security group IDs
from your cluster VPC. Separate multiple IDs with spaces. The security groups must allow traffic between compute nodes and the
cluster. For more information, see Security group requirements and considerations.
-
(Optional) Replace my-key-pair with the name of an Amazon EC2 key
pair for SSH connections to compute nodes. If you omit --ssh-key-name, compute
nodes don't have a key pair.
-
Replace arn:InstanceProfile with the ARN of your IAM
instance profile. If you don't have one prepared, see IAM instance profiles for AWS Parallel Computing Service.
-
Replace min-instances and
max-instances with integer values. Set the minimum (min) equal
to the maximum (max) for static capacity (for example, 5 min, 5 max). Set the minimum to 0
for fully dynamic scaling (for example, 0 min, 10 max). With Slurm 24.05 or
later, you can set the minimum greater than 0 and less than the maximum for mixed capacity.
This maintains a baseline number of instances and scales up as needed (for example, 2 min,
10 max).
-
Replace t3.large with the instance type to use. To use more
than one instance type, specify multiple instanceType settings separated by
spaces, for example --instance-configs instanceType=c6i.16xlarge
instanceType=c6a.16xlarge.
All instance types must have the same processor architecture (x86_64 or arm64) and
number of vCPUs. If the instance types have GPUs, all instance types must have the same
number of GPUs.
aws pcs create-compute-node-group --region region \
--cluster-identifier my-cluster \
--compute-node-group-name my-node-group \
--launch-template-mode MANAGED \
--ami-id ami-ExampleID1 \
--subnet-ids subnet-ExampleID1 \
--security-group-ids sg-ExampleID1 \
--ssh-key-name my-key-pair \
--iam-instance-profile-arn arn:InstanceProfile \
--scaling-configuration minInstanceCount=min-instances,maxInstanceCount=max-instances \
--instance-configs instanceType=t3.large
Example– Creating a compute node group with a custom launch template
Use a custom launch template if you need Amazon EC2 settings that aren't available as
instance settings, such as network interfaces or capacity reservations. You don't need to
specify --launch-template-mode because CUSTOM is the default.
Replace lt-ExampleID1 with the ID of your launch template, and
launch-template-version with a specific version of it.
AWS PCS associates your compute node group with that version of the launch
template.
If you don't have a launch template, see Using Amazon EC2 launch templates with AWS PCS. Don't use a launch template named
pcs-identifier-do-not-delete. AWS PCS creates
launch templates with this name for its own use. For more information, see Launch templates that AWS PCS creates.
aws pcs create-compute-node-group --region region \
--cluster-identifier my-cluster \
--compute-node-group-name my-node-group \
--subnet-ids subnet-ExampleID1 \
--custom-launch-template id=lt-ExampleID1,version='launch-template-version' \
--iam-instance-profile-arn arn:InstanceProfile \
--scaling-configuration minInstanceCount=min-instances,maxInstanceCount=max-instances \
--instance-configs instanceType=t3.large
With a custom launch template, --ami-id,
--security-group-ids, and --ssh-key-name are optional. For each
option that you specify, AWS PCS uses your value instead of the value in the launch
template. For each option that you omit, AWS PCS uses the value in the launch template.
For example, suppose that you add --security-group-ids
sg-ExampleID2 to this command. Compute nodes use only
sg-ExampleID2, not the security groups in the launch template.
They still use the AMI and key pair in the launch template. If neither the command nor
the launch template has an AMI or security groups, AWS PCS rejects the request. You
can't specify --security-group-ids if your launch template defines network
interfaces. For more information, see How AWS PCS chooses the AMI, security groups, and SSH key pair.
Example– Creating a compute node group with custom Slurm settings
aws pcs create-compute-node-group --region region \
--cluster-identifier my-cluster \
--compute-node-group-name my-node-group \
--launch-template-mode MANAGED \
--ami-id ami-ExampleID1 \
--subnet-ids subnet-ExampleID1 \
--security-group-ids sg-ExampleID1 \
--iam-instance-profile-arn arn:InstanceProfile \
--scaling-configuration minInstanceCount=min-instances,maxInstanceCount=max-instances \
--instance-configs instanceType=t3.large \
--slurm-configuration \
'slurmCustomSettings=[{parameterName=Features,parameterValue="gpu,nvme"}]'
For more information, see Custom Slurm settings for AWS PCS compute node groups.
Example– Creating a compute node group with a scale-down idle time
aws pcs create-compute-node-group --region region \
--cluster-identifier my-cluster \
--compute-node-group-name my-gpu-nodes \
--launch-template-mode MANAGED \
--ami-id ami-ExampleID1 \
--subnet-ids subnet-ExampleID1 \
--security-group-ids sg-ExampleID1 \
--iam-instance-profile-arn arn:InstanceProfile \
--scaling-configuration minInstanceCount=0,maxInstanceCount=10 \
--instance-configs instanceType=p4d.24xlarge \
--slurm-configuration scaleDownIdleTimeInSeconds=300
For nodes in this compute node group, AWS PCS uses this
scaleDownIdleTimeInSeconds value instead of the cluster-level value. This
setting requires
Slurm version 25.11 or later.
You can add the following optional settings to the
create-compute-node-group command.
-
Use --purchase-option to choose how AWS PCS purchases EC2 instances
for your compute node group. On-Demand is the default.
-
ONDEMAND – Use On-Demand Instances. Also choose this option
if you plan to use an On-Demand Capacity Reservation (ODCR). To use an ODCR, you need a
custom launch template that targets it.
For more information, see Using ODCRs with AWS PCS.
-
SPOT – Use Spot Instances. If you choose Spot Instances,
you can also use --spot-options
allocationStrategy=strategy to define
how AWS PCS chooses Spot capacity pools when it launches instances in the node group.
Valid values are lowest-price, capacity-optimized, and
price-capacity-optimized (the default).
For more information, see Allocation strategies for Spot Instances in the Amazon Elastic Compute Cloud User
Guide.
-
CAPACITY_BLOCK – Use an existing Amazon EC2 Capacity Blocks for ML
reservation. You need a custom launch template that targets the Capacity Block. For more
information, see Using Amazon EC2 Capacity Blocks for ML with AWS PCS.
-
INTERRUPTIBLE_CAPACITY_RESERVATION – Use a shared interruptible
ODCR (I-ODCR). You need a custom launch template that targets the I-ODCR. For more
information, see
Using I-ODCRs with AWS PCS.
-
Use --slurm-configuration to set Slurm options for the
nodes in the compute node group as custom Slurm settings. For example, set
Weight to set scheduling priority. Nodes with lower weights have higher
priority. Set RealMemory to set the memory on each node, for use with the
CR_CPU_Memory option. For the parameters that AWS PCS supports, see Custom Slurm settings for AWS PCS compute node groups.
-
Use --node-lifecycle-actions to run custom scripts at defined points in a
compute node's lifecycle, such as mounting file systems, installing software, or joining a
directory service. For more information, see Configure node lifecycle actions in AWS PCS. For example commands, see Node lifecycle action examples for AWS PCS.
-
Use --tags key1=value1,key2=value2
to add tags to the compute node group.
It can take several minutes to create the compute node group. To check its status, run the
following command. You can't add the compute node group to a queue until its status is
ACTIVE. For more information, see Creating a queue in AWS PCS.
aws pcs get-compute-node-group --region region \
--cluster-identifier my-cluster \
--compute-node-group-identifier my-node-group