couchdb unable to run on FIPS enabled CentOS

[Nanonid]

When FIPS mode is enabled, MD5 is disabled in OpenSSL.
Verify at the command prompt:

%openssl md5 [some file]
Error setting digest md5
140656009516960:error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips:digest.c:256:

Expected Behavior

Fallback to other message digest mechanism

Current Behavior

Running couchdb with FIPS enabled results in the following abort

md5_dgst.c(82): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
Aborted

Possible Solution

Provide alternative message digest mechanism. Remove direct calls to crypto:hash MD5, and reference couchdb hash.
There is an impact to the CouchDB API

Steps to Reproduce (for bugs)

1. Enable FIPS on CentOS 6.9 following this procedure
2. Build release couchdb
3. Execute couchdb
4. Couchdb aborts

Context

Unable to use CouchDB for a project. If CouchDB unable to run on FIPS compliant CentOS will have to abandon CouchDB.

Your Environment

• Version used: Couchdb as of commit d3a5a71
• Browser Name and version:
• Operating System and version (desktop or mobile): CentOS 6.9 yum update as of 2/15/2018
• Link to your project: Internal project

[Nanonid]

Error message:

[user@centos69 couchdb]$ rel/couchdb/bin/couchdb
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed

{"Kernel pid terminated",application_controller,"{application_start_failure,couch_epi,{{shutdown,{failed_to_start_child,"couch_epi|chttpd_auth|keeper",{notsup,[{crypto,notsup_to_error,1,[{file,"crypto.erl"},{line,831}]},{couch_epi_util,hash,1,[{file,"src/couch_epi_util.erl"},{line,25}]},{couch_epi_functions,data,1,[{file,"src/couch_epi_functions.erl"},{line,33}]},{couch_epi_module_keeper,do_reload_if_updated,1,[{file,"src/couch_epi_module_keeper.erl"},{line,116}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},{couch_epi_app,start,[normal,[]]}}}"}

[wohali]

Thanks @Nanonid .

As mentioned on IRC, CouchDB relies on MD5 hashes of documents for consistency guarantees. MD5 is not a FIPS-compliant hash algorithm, so technically CouchDB is not FIPS-compliant.

You may be able to work around this by using erlang's built in md5 function erlang:md5 instead of our call to crypto:hash(md5, ...) which calls out to openssl, causing the failure that you see. However, this is only a workaround that FIPS-mode RHEL doesn't know about - it doesn't actually make CouchDB FIPS-compliant. Only you can know if this is acceptable for your needs! :)

You could similarly do a search-and-replace on the md5 algorithm in the code and substitute one of the FIPS-compliant algorithms, such as SHA-512 or SHA-256 (with SHA-512 testing fastest on modern hardware, due to processor acceleration). However, this will break replication with other CouchDBs, as the _rev tokens and the Content-MD5 header will no longer match.

Improving our hash algorithm is something we'd like to do long-term, but obviously backwards compatibility needs to be thought out in greater detail. We'd probably need to add support for alternate hashing algorithms in a minor release, then move to a new default in a major release.

Big thanks to @rnewson for his brainstorming on this as well.

[Nanonid]

It seems the best path is to reference an internal couchdb hash function which enables a message digest internal function via configuration.
Since CouchDB is not using MD5 for authentication or authorization, it's not addressed by FIPS compliance requirements (AFAICT).
The issues is merely the direct linking to the OpenSSL MD5 hash, which could be worked around and not move off of MD5 for the time being.

Replacing the direct crypto:hash with a "digest" function which calls erlang:md5 perhaps based on compiler option would provide a work around for this situation without compatibility issues.

[Nanonid]

Of course, replacing explicitly with a non-cryptographic hash such as FarmHash would make good stable alternative.

[wohali]

@rnewson mentioned that there are (may be?) places where we still use md5 for security stuff, so the code needs a full audit before making a wholesale change to a non-cryptographic hash function. I'd also like to see performance measurements for FarmHash vs. SHA-256/512 since there is explicit processor acceleration for the latter; those benefits could outweigh those of a non-cryptographic solution.

[Nanonid]

Ideally there would be a benchmark for how erlang would call the routine, just in case there are OTP specific issues.

Acceleration could also vary across chips, and shouldn't necessarily be Intel only acceleration. Many different hashes can take advantage of AVX(1/2) on other CPU like Ryzen.

One of the benchmarks makes a good point that parallel hashing may also be more performant. This would then also change the calling semantics.

1. Parallel XXHash
2. Google HighwayHash Project
3. Node.js HighwayHash

[Nanonid]

t1ha hash seems very portable.

[wohali]

It also seems rather obscure...and I don't see any papers validating the algorithm, whereas obviously those exist for the hashes I listed.

Because of the linear nature of CouchDB operations on databases within the context of a single node, I don't think there is much opportunity for parallelization in our calling of hashing algorithms.

[Nanonid]

If we add a check for FIPS_mode() and fall back on erlang:MD5 if enabled, that would be least disturbing for non-FIPS users. Does this sound acceptable as a pull request? (only hypothesis right now)

[rnewson]

Hm not really. The function you want to call doesn’t exist in all the versions of Erlang we support.

…

Sent from my iPhone

On 28 Feb 2018, at 19:51, Nanonid ***@***.***> wrote: If we add a check for FIPS_mode() and fall back on erlang:MD5 if enabled, that would be least disturbing for non-FIPS users. Does this sound acceptable as a pull request? (only hypothesis right now) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Nanonid]

So we have a erlang:MD5 polyfill. Aren't there already such fills?
I'd rather not have a secure couchdb fork, with IA patches, but it is necessary if the master can't be made compliant.

[rnewson]

I was referring the FIPS_mode() function.

[nickva]

Like @rnewson mentioned it is about the FIPS check.

erlang:md5 seems to be there in every supported Erlang version R16B03 -> 20.0.

crypto:info_fips() is a newer function and only present since 20.0 http://erlang.org/doc/man/crypto.html#info_fips-0

At first thought it would seem why not just switch all uses of md5 to erlang:md5 but it turns out it can be quite a bit slower for large values. For 4MB binaries for example:

3> f(Bin), Bin = crypto:strong_rand_bytes(1 bsl 22).
<<93,...>>

4> timer:tc(fun() -> crypto:hash(md5, Bin) end ).
{6929, <<75,...>>}

5> timer:tc(fun() -> crypto:hash(md5, Bin) end ).
{7064, <<75,...>>}

6> timer:tc(fun() -> erlang:md5(Bin) end ).
{40849, <<75,...>>}

7> timer:tc(fun() -> erlang:md5(Bin) end ).
{85613, <<75,...>>}

(The result of timer:tc/1 is {Microseconds, FunResult})

Maybe you'd want to move the md5 calculation to couch_util and provide a macro can be used at compile to to pick between the implementations. If FIPS_MODE enabled you'd pick erlang:md5 otherwise use the other one.

We recently did a similar macro thing for some of random functions:

https://github.com/apache/couchdb/blob/75984da4b22003d0e46a9fe1001978d999387636/src/couch/src/couch_rand.erl

couchdb/src/couch/rebar.config.script

Line 138 in a99cc6f

{platform_define, "^17", 'NORANDMODULE'},

[NatTuck]

Any use of md5 in 2018 is pretty serious code smell. There's a reason FIPS mode straight up removes the function.