Release 3.5 - 02/21/23-07:45am PST

Author: Kari Stromsland

doc_source/HeadNode-v3.md

@@ -448,5 +448,5 @@ For example, if a custom AMI has an encrypted snapshot associated with it, the f
      ]
   }
 ```
-To troubleshoot custom AMI validation warnings, see [Troubleshooting custom AMI issues](troubleshooting-v3.md#troubleshooting-v3-custom-amis)\.  
+To troubleshoot custom AMI validation warnings, see [Troubleshooting custom AMI issues](troubleshooting-v3-custom-amis.md)\.  
 [Update policy: If this setting is changed, the update is not allowed.](using-pcluster-update-cluster-v3.md#update-policy-fail-v3)

doc_source/Image-v3.md

@@ -40,5 +40,5 @@ For example, if a custom AMI has an encrypted snapshot associated with it, the f
    ]
 }
 ```
-To troubleshoot custom AMI validation warnings, see [Troubleshooting custom AMI issues](troubleshooting-v3.md#troubleshooting-v3-custom-amis)\.  
+To troubleshoot custom AMI validation warnings, see [Troubleshooting custom AMI issues](troubleshooting-v3-custom-amis.md)\.  
 [Update policy: If this setting is changed, the update is not allowed.](using-pcluster-update-cluster-v3.md#update-policy-fail-v3)

doc_source/Monitoring-v3.md

@@ -4,7 +4,6 @@
 
 ```
 Monitoring:
-  DetailedMonitoring: boolean
   Logs:
     CloudWatch:
       Enabled: boolean
@@ -19,10 +18,6 @@ Monitoring:
 
 ## `Monitoring` properties<a name="Monitoring-v3.properties"></a>
 
-`DetailedMonitoring` \(**Optional**, `Boolean`\)  
-If `true`, detailed monitoring is enabled for all cluster nodes\. This enables 1 minute monitoring in the Amazon EC2 console\. The default value is `false`\.  
-[Update policy: If this setting is changed, the update is not allowed.](using-pcluster-update-cluster-v3.md#update-policy-fail-v3)
-
 `Logs` \(**Optional**\)  
 The log settings for the cluster\.  
 [Update policy: If this setting is changed, the update is not allowed.](using-pcluster-update-cluster-v3.md#update-policy-fail-v3)    

doc_source/Scheduling-v3.md

@@ -410,12 +410,12 @@ For example, suppose you define subnet\-1 and subnet\-2 for your queue\.
 `subnet-1` can be in AZ\-1 and `subnet-2` can be in AZ\-2\.  
 If you configure only one instance type and want to use multiple subnets, define your instance type in `Instances` rather than `InstanceType`\.  
 For example, define `ComputeResources` / `Instances` / `InstanceType`=`instance.type` instead of `ComputeResources` / `InstanceType`=`instance.type`\.  
+Elastic Fabric Adapter \(EFA\) isn't supported over different availability zones\.
 The use of multiple Availability Zones might cause increases in storage networking latency and added inter\-AZ data transfer costs\. For example, this could occur when an instance accesses file storage that's located in a different AZ\. For more information, see [Data Transfer within the same AWS Region](https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer_within_the_same_AWS_Region)\.  
 
 **Cluster updates to change from the use of a single subnet to multiple subnets:**
 + Suppose the subnet definition of a cluster is defined with a single subnet and an AWS ParallelCluster managed FSx for Lustre file system\. Then, you can't update this cluster with an updated subnet ID definition directly\. To make the cluster update, you must first change the managed file system to an external file system\. For more information, see [Convert AWS ParallelCluster managed storage to external storage](shared-storage-conversion-v3.md)\.
 + Suppose the subnet definition of a cluster is defined with a single subnet and an external Amazon EFS file system if EFS mount targets don't exist for all of the AZs for the multiple subnets defined to be added\. Then, you can't update this cluster with an updated subnet ID definition directly\. To make the cluster update or to create a cluster, you must first create all of the mount targets for all of the AZs for the defined multiple subnets\.
-  
 
 **Availability Zones and cluster capacity reservations defined in [CapacityReservationResourceGroupArn](#yaml-Scheduling-SlurmQueues-CapacityReservationResourceGroupArn):**
 + You can't create a cluster if there is no overlap between the set of instance types and availability zones covered by the defined capacity reservation resource group and the set of instance types and availability zones defined for the queue\.
@@ -528,7 +528,7 @@ For example, if a custom AMI has an encrypted snapshot associated with it, the f
    ]
 }
 ```
-To troubleshoot custom AMI validation warnings, see [Troubleshooting custom AMI issues](troubleshooting-v3.md#troubleshooting-v3-custom-amis)\.  
+To troubleshoot custom AMI validation warnings, see [Troubleshooting custom AMI issues](troubleshooting-v3-custom-amis.md)\.  
 [Update policy: The compute fleet must be stopped or QueueUpdateStrategy must be set for this setting to be changed for an update.](using-pcluster-update-cluster-v3.md#update-policy-queue-update-strategy-v3)
 
 #### `ComputeResources`<a name="Scheduling-v3-SlurmQueues-ComputeResources"></a>
@@ -582,6 +582,7 @@ For more information, see [Multiple instance type allocation with Slurm](slurm-m
 `Instances`:
    - `InstanceType`: string
 ```
+`EnableMemoryBasedScheduling` can't be enabled if you configure multiple instance types in [Instances](#yaml-Scheduling-SlurmQueues-ComputeResources-Instances)\.
 [Update policy: For this list values setting, a new value can be added during an update or the compute fleet must be stopped when removing an existing value.](using-pcluster-update-cluster-v3.md#update-policy-list-values-v3)    
 `InstanceType` \(**Required**, `String`\)  
 The instance type to use in this Slurm compute resource\. All of the instance types in a cluster must use the same processor architecture, either `x86_64` or `arm64`\.  
@@ -592,7 +593,9 @@ The instance types listed in [`Instances`](#yaml-Scheduling-SlurmQueues-ComputeR
 The instance types that are listed in [`Instances`](#yaml-Scheduling-SlurmQueues-ComputeResources-Instances) can have:  
 + Different amount of memory\.
 
-  In this case, the minimum memory is to be set as a consumable Slurm resource\. [`EnableMemoryBasedScheduling`](#yaml-Scheduling-SlurmSettings-EnableMemoryBasedScheduling) can't be enabled for multiple instance types\.
+  In this case, the minimum memory is to be set as a consumable Slurm resource\.
+
+  If you specify multiple instance types, `EnableMemoryBasedScheduling` can't be enabled\.
 + Different network cards\.
 
   In this case, the number of network interfaces configured for the compute resource is defined by the instance type with the smallest number of network cards\.
@@ -640,6 +643,7 @@ Efa:
 `Enabled` \(**Optional**, `Boolean`\)  
 Specifies that Elastic Fabric Adapter \(EFA\) is enabled\. To view the list of EC2 instances that support EFA, see [Supported instance types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html#efa-instance-types) in the *Amazon EC2 User Guide for Linux Instances*\. For more information, see [Elastic Fabric Adapter](efa-v3.md)\. We recommend that you use a cluster [`SlurmQueues`](#Scheduling-v3-SlurmQueues) / [`Networking`](#Scheduling-v3-SlurmQueues-Networking) / [`PlacementGroup`](#yaml-Scheduling-SlurmQueues-Networking-PlacementGroup) to minimize latencies between instances\.  
 The default value is `false`\.  
+Elastic Fabric Adapter \(EFA\) isn't supported over different availability zones\. For more information, see [SubnetIds](#yaml-Scheduling-SlurmQueues-Networking-SubnetIds)\.
 If you're defining a custom security group in [SecurityGroups](#yaml-Scheduling-SlurmQueues-Networking-SecurityGroups), make sure that your EFA\-enabled instances are members of a security group that allows all inbound and outbound traffic to itself\.
 [Update policy: The compute fleet must be stopped or QueueUpdateStrategy must be set for this setting to be changed for an update.](using-pcluster-update-cluster-v3.md#update-policy-queue-update-strategy-v3)  
 `GdrSupport` \(**Optional**, `Boolean`\)  
@@ -940,6 +944,7 @@ The default value is `false`\.
 Enabling memory\-based scheduling impacts the way that the Slurm scheduler handles jobs and node allocation\.  
 For more information, see [Slurm memory\-based scheduling](slurm-mem-based-scheduling-v3.md)\.
 `EnableMemoryBasedScheduling` is supported starting with AWS ParallelCluster version 3\.2\.0\.
+`EnableMemoryBasedScheduling` can't be enabled if you configure multiple instance types in [Instances](#yaml-Scheduling-SlurmQueues-ComputeResources-Instances)\.
 [Update policy: The compute fleet must be stopped for this setting to be changed for an update.](using-pcluster-update-cluster-v3.md#update-policy-compute-fleet-v3)
 
 ### `Database`<a name="Scheduling-v3-SlurmSettings-Database"></a>

doc_source/api-reference-v3.md

@@ -30,12 +30,12 @@ The template used to deploy the API is available at the following URL:
 https://<REGION>-aws-parallelcluster.s3.<REGION>.amazonaws.com/parallelcluster/<VERSION>/api/parallelcluster-api.yaml
 ```
 
-where `<REGION>` is the AWS Region where the API needs to be deployed to and `<VERSION>` is the AWS ParallelCluster version \(e\.g\. 3\.4\.1\)\. 
+where `<REGION>` is the AWS Region where the API needs to be deployed to and `<VERSION>` is the AWS ParallelCluster version \(e\.g\. 3\.5\.0\)\. 
 
 The [Docker](https://aws.amazon.com/docker/) image used to deploy the AWS Lambda function implementing AWS ParallelCluster features is available at:  [https://gallery\.ecr\.aws/parallelcluster/pcluster\-api](https://gallery.ecr.aws/parallelcluster/pcluster-api) 
 
 **Warning**  
-Any user in the AWS account, that has privileged access to AWS Lambda or Amazon API Gateway services, will automatically inherit permissions to administer AWS ParallelCluster API resources\.
+Any user in the AWS account, that has privileged access to AWS Lambda or Amazon API Gateway services, automatically inherits permissions to administer AWS ParallelCluster API resources\.
 
 ## Deploy with AWS CLI<a name="api-reference-deploy-v3"></a>
 
@@ -50,7 +50,7 @@ Run the following commands to deploy the API
 ```
 $ REGION=<region>
 $ API_STACK_NAME=<stack-name>  # This can be any name
-$ VERSION=3.4.1
+$ VERSION=3.5.0
 $ aws cloudformation create-stack \
     --region ${REGION} \
     --stack-name ${API_STACK_NAME} \
@@ -96,7 +96,7 @@ The `ParallelClusterApiUserRole` has permission to invoke all AWS ParallelClus
 ```
 $ REGION=<region>
 $ API_STACK_NAME=<stack-name>  # This needs to correspond to the existing API stack name
-$ VERSION=3.4.1
+$ VERSION=3.5.0
 $ aws cloudformation update-stack \
     --region ${REGION} \
     --stack-name ${API_STACK_NAME} \

doc_source/autoscaling.md

@@ -12,7 +12,7 @@ Clusters deployed with AWS ParallelCluster are elastic in several ways\. Setting
 
 ## Scaling up<a name="scaling-up"></a>
 
-Every minute, a process called [https://github.com/aws/aws-parallelcluster-node/tree/release-2.11/src/jobwatcher](https://github.com/aws/aws-parallelcluster-node/tree/release-2.11/src/jobwatcher) runs on the head node\. It evaluates the current number of instances required by the pending jobs in the queue\. If the total number of busy nodes and requested nodes is greater than the current desired value in the Auto Scaling group, it adds more instances\. If you submit more jobs, the queue is re\-evaluated and the Auto Scaling group is updated, up to the specified [`max_queue_size`](cluster-definition.md#configuration-max-queue-size)\.
+Every minute, a process called [https://github.com/aws/aws-parallelcluster-node/tree/v2.11.4/src/jobwatcher](https://github.com/aws/aws-parallelcluster-node/tree/v2.11.4/src/jobwatcher) runs on the head node\. It evaluates the current number of instances required by the pending jobs in the queue\. If the total number of busy nodes and requested nodes is greater than the current desired value in the Auto Scaling group, it adds more instances\. If you submit more jobs, the queue is re\-evaluated and the Auto Scaling group is updated, up to the specified [`max_queue_size`](cluster-definition.md#configuration-max-queue-size)\.
 
 With an SGE scheduler, each job requires a number of slots to run \(one slot corresponds to one processing unit, for example, a vCPU\)\. To evaluate the number of instances that are required to serve the currently pending jobs, the `jobwatcher` divides the total number of requested slots by the capacity of a single compute node\. The capacity of a compute node that corresponds to the number of available vCPUs depends on the Amazon EC2 instance type that's specified in the cluster configuration\.
 
@@ -27,7 +27,7 @@ In this example, the `jobwatcher` requires three new compute instances in the Au
 
 ## Scaling down<a name="scaling-down"></a>
 
-On each compute node, a process called [https://github.com/aws/aws-parallelcluster-node/tree/release-2.11/src/nodewatcher](https://github.com/aws/aws-parallelcluster-node/tree/release-2.11/src/nodewatcher) runs and evaluates the idle time of the node\. An instance is terminated when both of the following conditions are met: 
+On each compute node, a process called [https://github.com/aws/aws-parallelcluster-node/tree/v2.11.4/src/nodewatcher](https://github.com/aws/aws-parallelcluster-node/tree/v2.11.4/src/nodewatcher) runs and evaluates the idle time of the node\. An instance is terminated when both of the following conditions are met: 
 + An instance has no jobs for a period of time longer than the [`scaledown_idletime`](scaling-section.md#scaledown-idletime) \(the default setting is 10 minutes\)
 + There are no pending jobs in the cluster
 

doc_source/building-custom-ami-v3.md

@@ -55,7 +55,7 @@ Steps:
       "cloudformationStackStatus": "CREATE_IN_PROGRESS",
       "cloudformationStackArn": "arn:aws:cloudformation:us-east-1:123456789012:stack/IMAGE_ID/abcd1234-ef56-gh78-ij90-1234abcd5678",
       "region": "us-east-1",
-      "version": "3.4.1"
+      "version": "3.5.0"
     }
    }
    ```
@@ -78,18 +78,18 @@ Steps:
    # BEFORE COMPLETE
    {
     "imageConfiguration": {
-      "url": "https://parallelcluster-1234abcd5678efgh-v1-do-not-delete.s3.amazonaws.com/parallelcluster/3.4.1/images/IMAGE_ID-abcd1234efgh5678/configs/image-config.yaml?...",
+      "url": "https://parallelcluster-1234abcd5678efgh-v1-do-not-delete.s3.amazonaws.com/parallelcluster/3.5.0/images/IMAGE_ID-abcd1234efgh5678/configs/image-config.yaml?...",
     },
     "imageId": "IMAGE_ID",
     "imagebuilderImageStatus": "BUILDING",
     "imageBuildStatus": "BUILD_IN_PROGRESS",
     "cloudformationStackStatus": "CREATE_IN_PROGRESS",
     "cloudformationStackArn": "arn:aws:cloudformation:us-east-1:123456789012:stack/IMAGE_ID/abcd1234-ef56-gh78-ij90-1234abcd5678",
     "region": "us-east-1",
-    "version": "3.4.1",
+    "version": "3.5.0",
     "cloudformationStackTags": [
       {
-        "value": "3.4.1",
+        "value": "3.5.0",
         "key": "parallelcluster:version"
       },
       {
@@ -105,7 +105,7 @@ Steps:
    # AFTER COMPLETE
    {
     "imageConfiguration": {
-      "url": "https://parallelcluster-1234abcd5678efgh-v1-do-not-delete.s3.us-east-1.amazonaws.com/parallelcluster/3.4.1/images/IMAGE_ID-abcd1234efgh5678/configs/image-config.yaml?Signature=..."
+      "url": "https://parallelcluster-1234abcd5678efgh-v1-do-not-delete.s3.us-east-1.amazonaws.com/parallelcluster/3.5.0/images/IMAGE_ID-abcd1234efgh5678/configs/image-config.yaml?Signature=..."
     },
     "imageId": "IMAGE_ID",
     "imageBuildStatus": "BUILD_COMPLETE",
@@ -124,7 +124,7 @@ Steps:
         ],
       "architecture": "x86_64"
     },
-    "version": "3.4.1"      
+    "version": "3.5.0"      
    }
    ```
 
@@ -146,10 +146,10 @@ After running the [`build-image`](pcluster.build-image-v3.md) command, it's poss
 $ pcluster get-image-stack-events --image-id IMAGE_ID --region REGION --query "events[0]"
 {
  "eventId": "ParallelClusterImage-CREATE_IN_PROGRESS-2022-04-05T21:39:24.725Z",
- "physicalResourceId": "arn:aws:imagebuilder:us-east-1:123456789012:image/parallelclusterimage-IMAGE_ID/3.4.1/1",
+ "physicalResourceId": "arn:aws:imagebuilder:us-east-1:123456789012:image/parallelclusterimage-IMAGE_ID/3.5.0/1",
  "resourceStatus": "CREATE_IN_PROGRESS",
  "resourceStatusReason": "Resource creation Initiated",
- "resourceProperties": "{\"InfrastructureConfigurationArn\":\"arn:aws:imagebuilder:us-east-1:123456789012:infrastructure-configuration/parallelclusterimage-abcd1234-ef56-gh78-ij90-1234abcd5678\",\"ImageRecipeArn\":\"arn:aws:imagebuilder:us-east-1:123456789012:image-recipe/parallelclusterimage-IMAGE_ID/3.4.1\",\"DistributionConfigurationArn\":\"arn:aws:imagebuilder:us-east-1:123456789012:distribution-configuration/parallelclusterimage-abcd1234-ef56-gh78-ij90-1234abcd5678\",\"Tags\":{\"parallelcluster:image_name\":\"IMAGE_ID\",\"parallelcluster:image_id\":\"IMAGE_ID\"}}",
+ "resourceProperties": "{\"InfrastructureConfigurationArn\":\"arn:aws:imagebuilder:us-east-1:123456789012:infrastructure-configuration/parallelclusterimage-abcd1234-ef56-gh78-ij90-1234abcd5678\",\"ImageRecipeArn\":\"arn:aws:imagebuilder:us-east-1:123456789012:image-recipe/parallelclusterimage-IMAGE_ID/3.5.0\",\"DistributionConfigurationArn\":\"arn:aws:imagebuilder:us-east-1:123456789012:distribution-configuration/parallelclusterimage-abcd1234-ef56-gh78-ij90-1234abcd5678\",\"Tags\":{\"parallelcluster:image_name\":\"IMAGE_ID\",\"parallelcluster:image_id\":\"IMAGE_ID\"}}",
  "stackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/IMAGE_ID/abcd1234-ef56-gh78-ij90-1234abcd5678",
  "stackName": "IMAGE_ID",
  "logicalResourceId": "ParallelClusterImage",
@@ -164,11 +164,11 @@ After about 15 minutes, the stack events appear in the log event entry related t
 $ pcluster list-image-log-streams --image-id IMAGE_ID --region REGION \
     --query 'logStreams[*].logStreamName'
 
- "3.4.1/1"
+ "3.5.0/1"
 ]
 
 $ pcluster get-image-log-events --image-id IMAGE_ID --region REGION \
---log-stream-name 3.4.1/1 --limit 3
+--log-stream-name 3.5.0/1 --limit 3
 {
  "nextToken": "f/36295977202298886557255241372854078762600452615936671762",
  "prevToken": "b/36295977196879805474012299949460899222346900769983430672",
@@ -178,7 +178,7 @@ $ pcluster get-image-log-events --image-id IMAGE_ID --region REGION \
      "timestamp": "2022-04-05T22:13:26.633Z"
    },
    {
-     "message": "Document arn:aws:imagebuilder:us-east-1:123456789012:component/parallelclusterimage-test-abcd1234-ef56-gh78-ij90-1234abcd5678/3.4.1/1",
+     "message": "Document arn:aws:imagebuilder:us-east-1:123456789012:component/parallelclusterimage-test-abcd1234-ef56-gh78-ij90-1234abcd5678/3.5.0/1",
      "timestamp": "2022-04-05T22:13:26.741Z"
    },
    {
@@ -195,7 +195,7 @@ Continue to check with the [`describe-image`](pcluster.describe-image-v3.md) com
 $ pcluster describe-image --image-id IMAGE_ID --region REGION
 {
  "imageConfiguration": {
-   "url": "https://parallelcluster-1234abcd5678efgh-v1-do-not-delete.s3.us-east-1.amazonaws.com/parallelcluster/3.4.1/images/IMAGE_ID-abcd1234efgh5678/configs/image-config.yaml?Signature=..."
+   "url": "https://parallelcluster-1234abcd5678efgh-v1-do-not-delete.s3.us-east-1.amazonaws.com/parallelcluster/3.5.0/images/IMAGE_ID-abcd1234efgh5678/configs/image-config.yaml?Signature=..."
  },
  "imageId": "IMAGE_ID",
  "imageBuildStatus": "BUILD_COMPLETE",
@@ -214,7 +214,7 @@ $ pcluster describe-image --image-id IMAGE_ID --region REGION
      ],
    "architecture": "x86_64"
  },
- "version": "3.4.1"      
+ "version": "3.5.0"      
 }
 ```