[PM-18100] Add mlock and memfd_secret implementations

[dani-garcia]

🎟️ Tracking

https://bitwarden.atlassian.net/browse/PM-18100

📔 Objective

Implement some new backends for the KeyStore that use the OS memory protections:

• memfd_secret: On newer Linux only, but should protect any content in the store from being read from external programs, even if running as root. It also provides some minor protection against malicious kernel modules, but that can be bypassed.
• mlock: Supported on Unix. Doesn't provide any real protection from reads, but ensures that the keys aren't swapped to disk. Note that Windows also supports a similar VirtualLock API, but I've had some trouble with the crates depending on an older windows crate, so it's not enabled for Windows yet.

Note that for this implementation to be useful, we need to revert the boxing of keys so they are stack allocated. This would mean either not exposing them publicly, or having a separate internal key type.

This requires a MSRV bump to 1.80 for the use of LazyLock, but #256 requires 1.82 anyway.

For this implementation we're using the hashbrown crate, which allows us to create a HashMap that uses a custom allocator. I've implemented a custom allocator for the two protections mentioned above.

To ensure that this works correctly, I've implemented a simple fuzzing test that executes thousands of random operations and compares the results and contents of these new backeds against the basic rust hashmap that we're using now.

⏰ Reminders before review

• Contributor guidelines followed
• All formatters and local linters executed and passed
• Written new unit and / or integration tests where applicable
• Protected functional changes with optionality (feature flags)
• Used internationalization (i18n) for all UI strings
• CI builds passed
• Communicated to DevOps any deployment requirements
• Updated any necessary documentation (Confluence, contributing docs) or informed the documentation
  team

🦮 Reviewer guidelines

• 👍 (:+1:) or similar for great changes
• 📝 (:memo:) or ℹ️ (:information_source:) for notes or general info
• ❓ (:question:) for questions
• 🤔 (:thinking:) or 💭 (:thought_balloon:) for more open inquiry that's not quite a confirmed
  issue and could potentially benefit from discussion
• 🎨 (:art:) for suggestions / improvements
• ❌ (:x:) or ⚠️ (:warning:) for more significant problems or concerns needing attention
• 🌱 (:seedling:) or ♻️ (:recycle:) for future improvements or indications of technical debt
• ⛏ (:pick:) for minor or nitpick changes

[github-actions]

Checkmarx One – Scan Summary & Details – 64eb910d-4eba-4c8f-b899-d8a02da34d90

Great job, no security vulnerabilities found in this Pull Request

[codecov]

Codecov Report

Attention: Patch coverage is 84.21053% with 33 lines in your changes missing coverage. Please review.

Project coverage is 71.89%. Comparing base (c3e3dac) to head (01925f7).

Files with missing lines	Patch %	Lines
...den-crypto/src/store/backend/implementation/mod.rs	85.41%	14 Missing ⚠️
.../implementation/custom_alloc/linux_memfd_secret.rs	78.00%	11 Missing ⚠️
...tore/backend/implementation/custom_alloc/malloc.rs	77.77%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #125      +/-   ##
==========================================
+ Coverage   71.77%   71.89%   +0.12%     
==========================================
  Files         224      227       +3     
  Lines       18403    18610     +207     
==========================================
+ Hits        13208    13379     +171     
- Misses       5195     5231      +36     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[addisonbeck]

This looks fine to me, but I am definitely not the perfect person for review. I'll let @Hinton take a look before approving. Maybe even @coroiu taking a look would be nice.

[coroiu]

@dani-garcia In the description you say

Note that for this implementation to be useful, we need to revert the boxing of keys so they are stack allocated. This would mean either not exposing them publicly, or having a separate internal key type.

Does this mean that we need to start using the KeyRef traits? Why can't these be exposed publicly?

[coroiu]

question: what happens if it isn't? What is returned as ptr if memfd_secret isn't supported? Trying to write to an invalid pointer/pointer to memory outside of the process should cause a segmentation fault, right?

[dani-garcia]

In theory it shouldn't happen, the memfd_secret_sized function should return None if it can't allocate or the feature is not supported, and if it returns Some, then the file descripto is valid and open for read/write according to the API: https://man7.org/linux/man-pages/man2/memfd_secret.2.html#DESCRIPTION.

This was added more as a sanity check during development, so it can probably be removed.

[coroiu]

question: is aligned or isn't aligned?

[dani-garcia]

I tried to mean validate that the pointer is aligned, return an error if not but I can see how that's a bit confusing.

So the case inside the if, we check if it's not aligned to return an error, and only continue after the if when the pointer is valid. I'll try to rewrite the comment to be more clear.

[coroiu]

Ohh yeah ok that makes sense, interesting how that can be interpreted differently :D Thanks!

[dani-garcia]

Updated to be a bit less confusing and also included more detail.

sdk-internal/crates/bitwarden-crypto/src/store/backend/implementation/custom_alloc/linux_memfd_secret.rs

Lines 58 to 63 in 01925f7

	// The pointer that we return needs to be aligned to the requested alignment.
	// If that's not the case, we should free the memory and return an allocation error.
	// While we check for this error condition, this should never happen in practice, as the
	// pointer returned by `memfd_secret_sized` should be page-aligned (typically 4KB)
	// which should be larger than any possible alignment value.
	if (ptr.as_ptr() as *mut u8).align_offset(layout.align()) != 0 {

[dani-garcia]

@dani-garcia In the description you say

Note that for this implementation to be useful, we need to revert the boxing of keys so they are stack allocated. This would mean either not exposing them publicly, or having a separate internal key type.

Does this mean that we need to start using the KeyRef traits? Why can't these be exposed publicly?

So what these memory protection schemes do is protect a region of memory where we place the keys. In this case inside a hashmap. The reason why we apply the protection to the hashmap and not the individual keys is that some of these operations are slow, and others require page alignment, which would be wasteful if we did it per key.

The problem is that if the keys are not stack allocated, then the only thing we're protecting is the pointers to the keys, as the keys would be allocated somewhere else. If you check the definition of our keys, they are all boxed inside:

struct Aes256CbcKey {
     enc_key: Pin<Box<GenericArray<u8, U32>>>,
}

This was done initially to avoid the keys being stack allocated, which makes it easier to accidentally make multiple copies in memory when moving the keys around, but it also invalidates the use of combined memory protections like these. If we remove this boxing to make the memfd_secret implementation useful, then that means we go back to the danger of having to be careful with the keys.

This leaves us with a few options if we want these protections to be effective:

• We remove any use of the keys outside the bitwarden-crypto, and then we can make them crate private, and just be careful with them in the few places that use them. The majority of the code would be dealing with KeyIds only. This is something we'd want to happen anyway, to avoid users having direct access to key material.
• We split the keys into two: pub(crate) InternalAes256CbcKey(GenericArray<u8, U32>) and pub Aes256CbcKey(Pin<Box<InternalAes256CbcKey>>) KeyStore only stores the internal variant, and the public variant is just a boxed wrapper.
• We just remove the boxing and keep the keys public, this would be easiest to implement, but it would mean backsliding a bit for the cases where we use the keys from non-crypto apis, which is still a few.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[quexten]

The reason why we apply the protection to the hashmap and not the individual keys is that some of these operations are slow

Just wondering, what does slow mean here? A few microseconds? Considering we don't create thousands of keys per second, this may be something to not worry about as much (if for some reason this requirement were an issue. I'm not saying we should).

[dani-garcia]

When I tested memfd_secret, it was much slower compared to a normal allocation, but still fast enough overall. I think the numbers were 5ish nanoseconds for a simple malloc vs 50-100ish microseconds for memfd_secret. This was tested in a fairly slow VM, so a real life machine is probably faster.

You're right that for the number of keys that we handle it's very unlikely that the operation cost or memory usage of the keys is going to be noticeable regardless of what we do. I think the main benefit is probably that centralizing the secure memory tricks to a single place makes them easier to reason about.

Another thing to have in mind that I didn't mention is that each use of memfd_secret counts as an open file descriptor towards the per-process file descriptor limit, which might cause a problem in some systems if we did per key allocations, and in some systems it still comes with a default of 1024 (though it's easy to increase).