Improve error handling when catalog contains binary data

[joshcooper]

Describe the Bug

If you accidentally include binary data in the catalog, then the problem is difficult to troubleshoot as the error doesn't specify which environment/module/resource/parameter is causing the issue.

Puppetserver 8 will fail compilation and its log will contain:

2024-02-14T20:26:59.001Z ERROR [qtp1784649573-50] [puppetserver] Puppet Server Error: Failed to serialize Puppet::Resource::Catalog for 'XXX': Could not render to Puppet::Network::Format[rich_data_json]: source sequence is illegal/malformed utf-8

which is surfaced on the agent as:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed to serialize ...

In puppet7, puppetserver will silently downgrade to PSON (though this can be disabled with allow_pson_serialization setting). The agent will deserialize the catalog as PSON, and then report a warning when trying to cache the catalog as JSON:

# puppet agent -t
...
Info: Unable to serialize catalog to json, retrying with pson. PSON is deprecated and will be removed in a future release

There are two variations to this problem. First, the string needs to be "labeled" as UTF-8, so String#encoding should return UTF-8 and not ASCII_8BIT (aka BINARY). Second, the string needs to be valid UTF-8, so String.valid_encoding? must be true. The latter case, can easily occur when using the file function instead of binary_file.

Expected Behavior

If binary data is accidentally introduced into the catalog and is not wrapped in Binary, as can occur when using the file function, then the compilation should fail indicating which resource caused the issue.

Steps to Reproduce

Steps to reproduce the behavior:

1. On puppetserver, run this script:

binary_content = "\xC0\xFF".force_encoding('binary')
File.binwrite('/tmp/src.bin', binary_content)

2. Create site.pp

# cat <<END > /etc/puppetlabs/code/environments/production/manifests/site.pp
file { '/tmp/dst.bin':
  ensure => file,
  content => file('/tmp/src.bin'),
}
END

3. Run the local agent, it will fail as described above depending on the agent and server versions.

Environment

• Puppet7 and 8

Additional Context

#9102
https://puppet.atlassian.net/browse/PUP-10096
https://puppetcommunity.slack.com/archives/C0W298S9G/p1707246678595899

[TobiPeterG]

I'm facing this exact issue.
I already tried to remove every non UTF8 file from my catalog using
grep -nP '[\x80-\xFF]' -r .
but I still get this error.
Is there any way to find out what causes it?

[github-actions]

Migrated issue to PUP-12092

[adamboutcher]

Migrated issue to PUP-12092

Helpfully this isnt publicly visible.

I have the same issue but cant figure a way to track this down to resolve it.

Related:
https://puppetcommunity.slack.com/archives/C0W298S9G/p1742478655280359
https://puppetcommunity.slack.com/archives/C0W298S9G/p1742479574842609

[joshcooper]

We are working on a fix for this in the puppet core repo and it should be released next month.

In the meantime, the usual culprit is attempting to inline the result of the file function in the catalog like

file { ''<dest path>":
  ensure  => file,
  content => file("<source path>")
}

If <source path> is contains binary content (like a DER-encoded RSA key, kerberos key tab file, java keystore, ...), then you will see the "Failed to serialize..." error.

To fix the issue, you'll want to do:

content => binary_file("<source path>")

Doing so will correctly wrap the String in a Binary data type.

[bastelfreak]

I got another interesting case that's hard to debug. Puppet agent on the CLI always seems to be fine:

[09:51:00] root@f-pup-rep-p01:~> puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Notice: Requesting catalog from puppet:8140
Notice: Catalog compiled by puppet.local
Info: Caching catalog for f-pup-rep-p01
Info: Applying configuration version '...'
Notice: Applied catalog in 19.84 seconds
[09:51:44] root@f-pup-rep-p01:~>

But when the puppet agent is running as a systemd unit, all runs seem to fail. From the foreman logs:

This setup has a single puppetserver and there were no code deployments. I assume this is caused by facts, which are different in an interactive shell vs running in a systemd unit?

Edit: I don't know how that happened, but restarting puppet.service on this node fixed it ¯_(ツ)_/¯

[bastelfreak]

To resurrect this from the slack history, this is a SQL statement from @Sharpie to find broken resources in PuppetDB

SELECT DISTINCT type, file, line, name AS parameter 
FROM resource_params AS p 
JOIN catalog_resources AS r ON p.resource = r.resource 
WHERE p.value ~ '\ufffd';