`.upsmap` binaries in `System.Speech`

[premun]

Context

The key goal of source-build is to satisfy the official packaging rules of commonly used Linux distributions, such as Fedora and Debian. Many Linux distributions have similar rules. These rules tend to have two main principles: consistent reproducibility, and source code for everything.

In order to support the "source code for everything" requirement, binary files are not allowed in product repositories. Aside from, binaries that can be created during the build process from source are better not to be checked in as one of the main goals of git is that humans can review the code changes.

Questions

• What scenario / which RID are these files used for?
• Are these files necessary for a successful build of the .NET SDK?
• If they are is, can they be removed from the repository and replaced with a source and process that synthesizes them during build?

Goal

We should comply with the source build requirements and get rid of these binaries. Based on the discussion here, it seems we might have following options:

• These can be serialized from json/xml https://github.com/dotnet/runtime/blob/main/src/libraries/System.Speech/src/Internal/AlphabetConverter.cs#L238-L258
• We can move this project into https://github.com/dotnet/maintenance-packages in case it's not part of the SDK.

Possible workarounds

At the moment, we only source-build Linux x64/arm64 so if this file is required for other RIDs, it can be temporarily removed from the source build. This is only in case it's difficult to replace the file with source. Other platforms will be supported by source build in the future though, so this problem will re-surface in case we go around it this way.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @ericstj, @jeffhandley
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Context

The key goal of source-build is to satisfy the official packaging rules of commonly used Linux distributions, such as Fedora and Debian. Many Linux distributions have similar rules. These rules tend to have two main principles: consistent reproducibility, and source code for everything.

In order to support the "source code for everything" requirement, binary files are not allowed in product repositories. Aside from, binaries that can be created during the build process from source are better not to be checked in as one of the main goals of git is that humans can review the code changes.

Questions

• What scenario / which RID are these files used for?
• Are these files necessary for a successful build of the .NET SDK?
• If they are is, can they be removed from the repository and replaced with a source and process that synthesizes them during build?

Goal

We should comply with the source build requirements and get rid of these binaries. Based on the discussion here, it seems we might have following options:

• These can be serialized from json/xml https://github.com/dotnet/runtime/blob/main/src/libraries/System.Speech/src/Internal/AlphabetConverter.cs#L238-L258
• We can move this project into https://github.com/dotnet/maintenance-packages in case it's not part of the SDK.

Possible workarounds

At the moment, we only source-build Linux x64/arm64 so if this file is required for other RIDs, it can be temporarily removed from the source build. This is only in case it's difficult to replace the file with source. Other platforms will be supported by source build in the future though, so this problem will re-surface in case we go around it this way.

Author:	premun
Assignees:	-
Labels:	untriaged, area-System.Speech
Milestone:	-

[ericstj]

We can move this project into https://github.com/dotnet/maintenance-packages in case it's not part of the SDK.

Can't we achieve the same without actually moving it? This library is only built for shipping as a nuget package and should not be consumed by any up-stack repositories. We can just omit it from source build. cc @ViktorHofer

[ViktorHofer]

Right, we already exclude other projects during source build and moving the project to another repo to solve this issue doesn't sound right.

[ViktorHofer]

Just add the project to these two places:

• https://github.com/dotnet/runtime/blob/main/src/libraries/oob-src.proj#L31-L33
• https://github.com/dotnet/runtime/blob/main/src/libraries/oob-all.proj#L22-L25

[premun]

Thanks! What would happen if I just cloaked it from the VMR so that it wouldn't be present? Is the project referenced somewhere or the behaviour is "build all .csprojs you can find"? Would it need to be rather kept but excluded through these proj files only with the binaries cloaked away?

[ViktorHofer]

Thanks! What would happen if I just cloaked it from the VMR so that it wouldn't be present?

Are such binaries prohibited, even if they are unused? Asking as excluding the ProjectReference to System.Speech from oob-src.proj and oob-all.proj will make them (.upsmap) unused.

[premun]

I think it's quite impossible for RedHat to tell which files are used during the actual build so I think it's reasonable for them to scan for binaries and expect that whatever is not used is not there.

They do have exemptions for test folders and other so the rules are quite blurry actually and of course you could technically put a binary in a test folder and use it from a regular build so yeah... But regardless of that, if we can omit binaries, that we don't use, from a repo, I think we just should?

[ViktorHofer]

I'm just concerned about the case in .NET 9 when the VMR build will replace the official build (including on Windows) and we suddenly can't build that library anymore. If those files are just an implementation detail and don't leak to the customer, we should just relace them with non binary formats (what you suggested as an option in the top post).

[ericstj]

I see it as somewhat general goodness to have a text sources for binary files since they're more human readable and change-trackable. I put together a quick tool that can reproduce the binary content from JSON as suggested.
https://gist.github.com/ericstj/6eae8007ba05fdf302c09baaa6ed8540

I'm not too happy with the verbosity of the json form, probably would prefer CSV or something similar. We might even be able to just convert to some static C# with lazily evaluated construction of these tables.

We discussed this in the VMR sync today and agreed that these binary resource files of well understood format shouldn't be blocking for our Linux partners - they haven't been so far. If that changes then we can add a build step like this that generates the binary form from text.