Skip to content

Conversation

@BenAtAmazon
Copy link
Contributor

@BenAtAmazon BenAtAmazon commented Sep 16, 2025

Proposed Changes

AWS EC2 API returns networkInterfaceSet and privateIpAddressesSet in arbitrary order, causing non-deterministic hostname resolution during peer discovery and inconsistent cluster formation when setting certain cluster_formation.aws.hostname_path configurations such as:

cluster_formation.aws.hostname_path = networkInterfaceSet,2,privateIpAddressesSet,1,privateIpAddress

See https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html

"The order of the elements in the response, including those within nested structures, might vary. Applications should not assume the elements appear in a particular order."

Changes:

  • Sort network interfaces by deviceIndex (0 first for primary ENI)
  • Sort private IP addresses by primary flag (primary=true first)
  • Add debug logging to show hostname path selection and sorting results (removed in second revision)
  • Add unit tests for sorting behavior

The sorting ensures deviceIndex=0 and primary=true IPs are consistently selected as intended, making peer discovery deterministic across deployments. Note that since EC2 does not provide an index for privateIpAddressesSet (like deviceIndex for networkInterfaceSet), this determinism for privateIpAddressesSet only applies if there are 1 or 2 members in privateIpAddressesSet via the primary=true or primary=false. If there was a 3rd member in privateIpAddressesSet, the two primary=false IPs orders are non-deterministic.

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

  • Bug fix (non-breaking change which fixes issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause an observable behavior change in existing systems)
  • Documentation improvements (corrections, new content, etc)
  • Cosmetic change (whitespace, formatting, etc)
  • Build system and/or CI

Checklist

Put an x in the boxes that apply.
You can also fill these out after creating the PR.
This is simply a reminder of what we are going to look for before merging your code.

  • Mandatory: I (or my employer/client) have have signed the CA (see https://github.com/rabbitmq/cla)
  • I have read the CONTRIBUTING.md document
  • I have added tests that prove my fix is effective or that my feature works
  • All tests pass locally with my changes
  • If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
  • If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

Further Comments

N/A

@michaelklishin michaelklishin added this to the 4.3.0 milestone Sep 16, 2025
@BenAtAmazon BenAtAmazon force-pushed the aws/fix-peer-discovery-hostname-path-ordering branch from e3d3637 to b209d6a Compare September 17, 2025 18:23
@SimonUnge
Copy link
Collaborator

Looks good to me!

AWS EC2 API returns networkInterfaceSet and privateIpAddressesSet in
arbitrary order, causing non-deterministic hostname resolution during
peer discovery. This leads to inconsistent cluster formation.

Changes:
- Sort network interfaces by deviceIndex (0 first for primary ENI)
- Sort private IP addresses by primary flag (primary=true first)
- Add debug logging to show hostname path selection and sorting results
- Add comprehensive unit tests for sorting behavior

The sorting ensures deviceIndex=0 and primary=true IPs are consistently
selected first, making peer discovery deterministic across deployments.
… feedback on debug logs and sorting helper functions)
@lukebakken lukebakken force-pushed the aws/fix-peer-discovery-hostname-path-ordering branch from b209d6a to 7cebf89 Compare September 23, 2025 19:46
@lukebakken lukebakken merged commit 4a32470 into rabbitmq:main Sep 23, 2025
285 checks passed
mergify bot pushed a commit that referenced this pull request Sep 23, 2025
* AWS peer discovery: ensure consistent hostname path ordering

AWS EC2 API returns networkInterfaceSet and privateIpAddressesSet in
arbitrary order, causing non-deterministic hostname resolution during
peer discovery. This leads to inconsistent cluster formation.

Changes:
- Sort network interfaces by deviceIndex (0 first for primary ENI)
- Sort private IP addresses by primary flag (primary=true first)
- Add debug logging to show hostname path selection and sorting results
- Add comprehensive unit tests for sorting behavior

The sorting ensures deviceIndex=0 and primary=true IPs are consistently
selected first, making peer discovery deterministic across deployments.

* AWS peer discovery: ensure consistent hostname path ordering (address feedback on debug logs and sorting helper functions)

(cherry picked from commit 4a32470)
michaelklishin added a commit that referenced this pull request Sep 24, 2025
AWS peer discovery: ensure consistent hostname path ordering (backport #14557)
michaelklishin pushed a commit that referenced this pull request Sep 24, 2025
* AWS peer discovery: ensure consistent hostname path ordering

AWS EC2 API returns networkInterfaceSet and privateIpAddressesSet in
arbitrary order, causing non-deterministic hostname resolution during
peer discovery. This leads to inconsistent cluster formation.

Changes:
- Sort network interfaces by deviceIndex (0 first for primary ENI)
- Sort private IP addresses by primary flag (primary=true first)
- Add debug logging to show hostname path selection and sorting results
- Add comprehensive unit tests for sorting behavior

The sorting ensures deviceIndex=0 and primary=true IPs are consistently
selected first, making peer discovery deterministic across deployments.

* AWS peer discovery: ensure consistent hostname path ordering (address feedback on debug logs and sorting helper functions)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants