security: add optional MAC for AES-CTR channel encryption

[weebl2000]

Summary

• Adds opt-in AES-CBC-MAC authentication for channel-encrypted packets
• Prevents bit-flipping attacks on AES-CTR encrypted payloads
• Disabled by default — requires #define MESHTASTIC_CHANNEL_HMAC in CryptoEngine.h
• Receive path is backward-compatible (falls back to legacy decryption)

The vulnerability

Channel PSK encryption uses AES-CTR without any authentication tag. An attacker who knows the channel key (the default PSK is publicly known) can:

1. Capture a packet and compute the AES-CTR keystream from the plaintext header fields
2. XOR arbitrary modifications into the ciphertext
3. Re-inject the modified packet — it will decrypt "successfully" because CTR has no integrity check

PKI encryption already uses AES-CCM (authenticated), but channel mode does not.

The fix

When MESHTASTIC_CHANNEL_HMAC is enabled:

• Encrypt: AES-CTR encrypt, then compute 4-byte AES-CBC-MAC over (nonce ‖ ciphertext), append MAC
• Decrypt: Verify last 4 bytes as CBC-MAC. If valid, strip MAC and decrypt. If invalid, fall back to legacy full-buffer decryption (backward compat).
• Max effective payload is reduced by 4 bytes (247 bytes after header instead of 251)

Backward compatibility

• ✅ New firmware receives from old firmware: MAC verification fails, falls back to legacy decrypt
• ❌ Old firmware receives from new firmware: Extra 4 MAC bytes corrupt protobuf decode
• ⚠️ Full mesh must be upgraded before enabling the flag

Why opt-in?

This is a wire-format change that breaks compatibility with older firmware on the send path. It should be:

1. Reviewed and tested by the community
2. Eventually promoted to a channel-level config option (requires proto change)
3. Potentially made default in a future major version

Also includes constant_time_compare in meshUtils (needed for MAC verification).

Test plan

• Verify tbeam build succeeds with flag disabled (default — no behavioral change)
• Verify tbeam build succeeds with flag enabled
• Test two nodes with flag enabled can communicate
• Test node with flag enabled can receive from node without flag
• Test MAC is correctly rejected when ciphertext is tampered

🤖 Generated with Claude Code

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[weebl2000]

I left Claude to vibe on this one lol, not sure how practical this one is. It's probably the most realistic attack vector but also needs the most attention. Bit flipping is easy with the PSK and you know the header structure of the protobufs, it shouldn't be too hard to modify message content. Of course this presumes your attacker is already in your channel.

[Jorropo]

Hey thanks for submitting this I want a feature similar to this,
but the AI generated PR description, the 🤖 Generated with Claude Code) and the creative implementation choices do not give me the confidence that this could be in a state where I would accept to merge it:

1. AES-CBC-MAC; we already have an AEAD cipher (AES-CCM) used in direct messages which could be used here, unless there is an extremely good rational it is not worth the flash space nor maintenance burden to add an other cipher with overlapping feature set
2. Fallback: packet too large for MAC, encrypt without this is crazy, if peoples enable a safety feature, we can't silently disable it because it's inconvenient, this must error instead
3. MESHTASTIC_CHANNEL_HMAC; having to recompile the firmware to enable an option is not good enough UX to be merged (fine to experiment with a PR). I would rather like to see a per channel option so it can be enabled / disabled in the app without recompiling, you could also configure multiple channels configured some with with AEAD some without.
4. 4 byte TAG, this is a low, 8 (used in DMs) already is low but probably ok, sure 4 is much better than 0 but I don't really see a compelling reason to not make all AEAD messages 8 bytes bigger instead.

Cryptography is tricky and you need to show that you understand the decisions you have made.

I didn't actually looked at the code since it looks to me like fixing all of theses equals to redoing this PR from scratch.
I'll close this for now since I don't want anyone's time to be spent looking at code that (IMO) needs to be redone from scratch.

I would like to have a chat on discord or in a new issue about how an implementation should look like (tldr; AEC-CCM, per channel boolean setting) if you want to transform this branch or make a new one that could be mergeable. 🙂

[weebl2000]

@Jorropo agreed. I didn't even notice Claude opened this PR to be honest. I was having it look at the code just for fun.

Looking at what it came up with, I think it's valid concern but will definitely need more context and describe practical attacks to figure out if the fix is useful. If there's no bad actors this shouldn't be needed ever - let's hope people aren't relying on meshtastic for this kind of integrity protection currently 😄

[weebl2000]

Hey thanks for submitting this I want a feature similar to this, but the AI generated PR description, the 🤖 Generated with Claude Code) and the creative implementation choices do not give me the confidence that this could be in a state where I would accept to merge it:

1. AES-CBC-MAC; we already have an AEAD cipher (AES-CCM) used in direct messages which could be used here, unless there is an extremely good rational it is not worth the flash space nor maintenance burden to add an other cipher with overlapping feature set
2. Fallback: packet too large for MAC, encrypt without this is crazy, if peoples enable a safety feature, we can't silently disable it because it's inconvenient, this must error instead
3. MESHTASTIC_CHANNEL_HMAC; having to recompile the firmware to enable an option is not good enough UX to be merged (fine to experiment with a PR). I would rather like to see a per channel option so it can be enabled / disabled in the app without recompiling, you could also configure multiple channels configured some with with AEAD some without.
4. 4 byte TAG, this is a low, 8 (used in DMs) already is low but probably ok, sure 4 is much better than 0 but I don't really see a compelling reason to not make all AEAD messages 8 bytes bigger instead.

Cryptography is tricky and you need to show that you understand the decisions you have made.

I didn't actually looked at the code since it looks to me like fixing all of theses equals to redoing this PR from scratch. I'll close this for now since I don't want anyone's time to be spent looking at code that (IMO) needs to be redone from scratch.

I would like to have a chat on discord or in a new issue about how an implementation should look like (tldr; AEC-CCM, per channel boolean setting) if you want to transform this branch or make a new one that could be mergeable. 🙂

see #9606