LSIF spec could use some extra clarity around embedded contents

[jasonmalinowski]

The LSIF spec states that the contents of a file included in an LSIF index is encoded base64:

It can be valuable to embed the contents of a document or project file into the dump as well. For example, if the content of the document is a virtual document generated from program meta data. The index format, therefore, supports an optional contents property on the document and project vertex. If used the content needs to be base64 encoded.

Given base64 is a encoding of a binary stream, this implies that there's a text encoding question. So some questions:

1. Should the binary stream be the raw file on disk, in whatever text encoded form it is? This then means it's the responsibility of any consumer to do encoding sniffing which may come to a different conclusion (and therefore different contents) than the indexer. The alternative is the indexer re-encodes in some preferred/specified text encoding prior to the base64 encoding, although that still creates other fun questions around binary file inputs to compilers.
2. For files that have no "native" encoding because the indexer generated them directly in memory, which encoding should be chosen as a preferred choice?

[jasonmalinowski]

And to be very clear: this is not a question about the encoding of the LSIF text stream itself (which could be encoded any number of ways), but rather the string that is being put into the base64 encoding that is then further embedded.

[dbaeumer]

We have seen problems with escaping and other things if we put the string content into the JSON. So the idea here is to have the content in utf8 but then encode it in base64 to make sure the JSON stays valid.

Does that make sense to you?

[jasonmalinowski]

The base64 makes sense, but I think this the bit I wanted:

have the content in utf8

So for text files being embedded, the preferred encoding is utf8, then base64 that, then stick it in the JSON? Or just in code, this:

https://github.com/dotnet/roslyn/pull/49306/files#diff-43f70920b65d3528875ab41476c2c3bcbc8eef12b93294abb1d4c5cd7a872ad0R102

Where UTF8 is preferred, even if the underlying file was encoded differently.