Improve RSA Key Compatibility, Public Key Extraction, and Pubsub Connection Handling

[sumanjeet0012]

What was wrong?

Issue #1106

The py-libp2p implementation had few issues which is causing problems during py-peer of universal connectivity interoperability:

1. RSA Key Compatibility:
2. Missing Public Key Extraction:
3. Pubsub Signature Validation Issues:
4. Pubsub Connection Management Problems:

How was it fixed?

1. Enhanced RSA Key Parsing

Added fallback parsing logic in libp2p/crypto/rsa.py that uses the cryptography library when PyCryptodome fails. When a PKIX-formatted key is encountered, it's converted to PKCS1 format that PyCryptodome can handle:

try:
    rsakey = RSA.import_key(key_bytes)
except ValueError:
    crypto_key = load_der_public_key(key_bytes, backend=default_backend())
    pkcs1_bytes = crypto_key.public_bytes(
        encoding=serialization.Encoding.DER,
        format=serialization.PublicFormat.PKCS1
    )
    rsakey = RSA.import_key(pkcs1_bytes)

2. Implemented Public Key Extraction

Added extract_public_key() method to the ID class in libp2p/peer/id.py that:

• Decodes the multihash from the peer ID
• Checks if it uses identity multihash (function code 0x00)
• Extracts and deserializes the embedded public key if present
• Returns None for peer IDs using SHA-256 hash (RSA and other large keys)

3. Updated Signature Validation Logic

Modified signature_validator() in libp2p/pubsub/validators.py to:

• First attempt to extract the public key from the sender's peer ID
• Fall back to using the msg.key field if extraction returns None
• Validate that the extracted or provided key matches the sender ID

This approach aligns with the libp2p specification and reduces message overhead for small keys.

4. Improved Pubsub Connection Management

Enhanced libp2p/pubsub/pubsub.py with:

• Duplicate detection: Check if a pubsub stream already exists before opening a new one
• Connection-aware peer removal: Only remove peers from pubsub when ALL their connections are closed, not just one
• Safe exception handling: Wrapped _handle_new_peer() in _handle_new_peer_safe() to prevent service crashes
• Active connection filtering: Use c.muxed_conn.is_closed to filter out closed connections when checking if a peer should be removed

All changes are fully backward compatible and maintain existing functionality while adding robustness and spec compliance.

To-Do

• Clean up commit history
• Add or update documentation related to these changes
• Add entry to the release notes

Cute Animal Picture

[sumanjeet0012]

@acul71 @seetadev
I have completed all the required changes for the py-peer of the Universal Connectivity DApp.

The UC DApp is currently functioning smoothly. The py-peer is now able to successfully connect to bootstrap nodes, establish connectivity with go-peer instances, and discover new peers using the Kademlia DHT.

[seetadev]

@sumanjeet0012 : This is an excellent PR — thank you for the clear analysis and solid execution. I am reviewing the changes and they significantly improve py-libp2p interoperability and robustness for the Universal Connectivity DApp.

The enhanced RSA key parsing with a safe fallback to cryptography for PKIX keys is well designed and fully backward compatible. Adding public key extraction from identity multihash Peer IDs aligns py-libp2p with the libp2p spec and enables cleaner, more efficient pubsub validation. The updated signature validation logic correctly prioritizes Peer ID–embedded keys and falls back to msg.key only when needed, improving both correctness and message efficiency.

The pubsub connection management improvements are particularly valuable: preventing duplicate streams, handling peer removal only when all connections are closed, safely wrapping peer handling to avoid crashes, and filtering out closed connections all make the system much more resilient in real-world network conditions.

On the same note, appreciate the work you’ve put into the py-peer for the Universal Connectivity DApp. It’s great to hear that everything is now functioning smoothly end-to-end. Successfully connecting to bootstrap nodes, interoperating with go-peer instances, and discovering peers via the Kademlia DHT are all key milestones indeed.

CCing @acul71 and @pacrob for their pointers, feedback and comments. Still reviewing the changes made.