Include armv7 symbols in boringssl vendoring

[orobio]

Updates the BoringSSL vendoring to include armv7 specific symbols.

I had to do a work-around that specifically checks out the version of BoringSSL that is currently used. This was necessary due to that the very latest on their main branch doesn't work properly with the current vendoring script. I suggest to not merge this until the vendoring script is updated to work with the latest BoringSSL version again, then the work-around can be removed.

The armv7 build is done using the following container: orobio/swift-armv7-cross-bullseye:5.10
It was created from orobio/swift-armv7.

This container is hosted on my DockerHub account. Would that be acceptable, or would you prefer to host it yourself?

Most of this is new territory for me, so feel free to mention anything that can be improved.

[Lukasa]

Sorry for letting this languish!

@shahmishal is there any way we can get armv7 Swift images upstream?

[orobio]

Any news on this? It would be nice to finish this up in one way or another. Let me know if there’s anything I can do!

[xtremekforever]

Maybe we just need to generate a Swift 6.0.1 unofficial container for armv7 now? Well, there are a couple of blockers to generating an SDK as was suggested previously:

• I looked at using the swift-sdk-generator to create an SDK but it's simply not ready to be able to generate an SDK for armv7 in its current state. It will require some updates, such as supporting Debian 11 or 12 to grab armv7 debs to generate an SDK. Then, it would need to be able to somehow grab the armv7 build from "somewhere" (maybe locally) and put the files in the correct place. Right now the SDK generator places target .swiftmodules and libraries in the swift.xctoolchain directory, which is not quite right since they should go into the SDK directory for the target!!!
• I had been previously investigating adding support into swiftlang/swift to cross compile for armv7, but ran into quite a few issues and a Swift runtime that didn't actually work properly on the target. Since then this effort has been paused due to other commitments on my end.

However, if we can just use some of the existing swift-armv7 scripts to build Swift 6.0.1 and generate a container, would that be enough to at least get this done? It would be simple enough to update what we have working and generate something that could be used.

[orobio]

Well, we have a working solution here, so unless there is any indication that this solution is not acceptable, I don’t think we need to do anything else.

Regarding the Swift version. This provides 5.10 for armv7, while the other architectures are still on 5.9, so I think that’s fine for now.

[Lukasa]

Yeah, I don't foresee an issue with the Swift version. If you can bring this up to current main I'm happy enough to merge this as-is.

[orobio]

@Lukasa : I rebased this on main. Some notes:

• I ran into an issue where running the script a second time causes an issue with the 5.9 containers (error: invalid tool type in 'tools' map). It looks like 5.10 leaves some data in the .build directory that 5.9 doesn't like. I didn't see this issue earlier when they were still on 5.8. Removing the .build directory will fix this, so it shouldn't be a big problem. If this is not acceptable (and there's no specific reason that they are still on 5.9), I can update the other containers to 5.10 as well.
• I'm running this on Linux and assume it would be difficult (impossible?) to get the MacOS and iOS builds working, so I disabled these locally. Could you please verify the complete script on your end?
• For the above reason I did not include the result of the script either. I'll leave it to you to either add the updated files to this branch before merging, or just take the new symbols in when you're updating BoringSSL in the future.

[xtremekforever]

@Lukasa what else needs to happen here to run the checks and get this merged? I am particularly curious because I've been building Swift for armv7 and I have it working on different distributions as well: https://github.com/xtremekforever/swift-armv7/releases

I am also looking at creating cross-compilation Swift SDKs for armv7, but to compile apps that depend on swift-crypto we need to manually provide the missing symbols for armv7 on a custom branch, like this one: swift-embedded-linux@87c004a

I know this is not an "official" container in this solution, but there are now more and more options to compile and run Swift on armv7 devices, so having to provide a branch with custom symbols like this seems a bit silly to have to do, especially as we already have a vendor script solution here.

Please let us know what else would be needed!!!

[Lukasa]

This is a good question, and I don't really know what the right answer is. I think I'd like to kick this over to the SSWG to discuss what the appropriate support policy is here. cc @FranzBusch @ktoso

[FranzBusch]

This is a good question, and I don't really know what the right answer is. I think I'd like to kick this over to the SSWG to discuss what the appropriate support policy is here. cc @FranzBusch @ktoso

My take on this is similar to other unsupported platforms like Android. Currently we only CI for supported platforms and the platform steering group is responsible for maintaining the list of supported platforms cc @al45tair.

Similarly for this PR I would suggest to land it without CI if we are happy to vendor these symbols in and treat it like Android for now. WDYT @Lukasa ?

[Lukasa]

Well I think the interesting question is actually the step regarding the symbol mangling, which will rely on an externally provided toolchain. We have done this in the past, but I think it'd be useful to clarify what the SSWG thinks is appropriate here.

[FranzBusch]

Okay I get it now. This PR relies on a docker image produced by @orobio to get the correct symbols. I don't think we can take that patch while relying on an unvetted docker image since it would be possible to compromise the security of our crypto primitives on armv7 platforms. I think at this point it is really a question for the platform steering group if we can get armv7 as an officially supported platform and get official docker images out. cc @al45tair and @shahmishal.

[xtremekforever]

Okay, so I guess if this is a security thing it makes more sense. So, our only shot is trying to get armv7 into Swift. I would so love this, but I can't do it by myself as I struggled so profusely with the build scripts and still did not manage to get something working.

Should this just be closed and then we can come back once there is official support for armv7 for Swift?

[Lukasa]

The issue is yours, and so you're free to do what you like with it. If you'd like to keep it open to track the request, I'm happy to leave it open, but if you'd feel better closing it we can do that too.

[orobio]

I don't think we can take that patch while relying on an unvetted docker image since it would be possible to compromise the security of our crypto primitives on armv7 platforms.

This pull request was created based on a discussion on issue #223 where an unofficial image was deemed acceptable. However, I do understand the security issues (and also asked in the description whether hosting on my Dockerhub account was a problem). It's great to finally have an answer, so thank you for that, but I do want to note that I wonder if this should've taken 9 months.

I think at this point it is really a question for the platform steering group if we can get armv7 as an officially supported platform and get official docker images out.

I think this would be great and I hope we can slowly and steadily continue working towards an official solution! Before we get there though, do you see any way in which we can provide a solution that will allow you to vet the docker image contents and also guarantee its integrity for repeated use?

[xtremekforever]

@Lukasa would you accept an alternate way of doing this instead of providing you an unvetted/uncontrolled container from Docker Hub? A few ideas:

1. Create a container in swift-docker perhaps? We could potentially provide a Dockerfile that could build a container that contains an armv7 SDK from here, and that's something that could be vetted by the swiftlang team. If needed it could even be taken a step further and simply perform the cross-compilation of Swift for armv7 from the swift-docker repo to build the container- which would only be a small set of scripts to accomplish.
2. The other way I'd like to suggest that could be a lot simpler is to provide a script that can invoke the swift-sdk-generator to build the Swift SDK for armv7, as suggested in the original ticket. It's only a small set of commands, and the swift-sdk-generator is more trusted since it pulls Ubuntu or Debian sources from known places.
3. Finally, if none of these would be acceptable- what about simply hardcoding these symbols right into swift-crypto just for armv7? What would be the downside or issue with this if it was done this way?

[Lukasa]

I'm open to considering a script that invokes the SDK generator. As you've suggested, these are easier to audit at the point of use.

[xtremekforever]

@Lukasa is it okay to use a Swift 6.x version as a minimum, or should the Swift version be 5.10 to match the Docker container in the vendoring script?

Also, by the logic of generating an SDK, it may be even easier to generate a Docker container from a script (with the Dockerfile included) as an alternative to generating the Swift SDK. It would be easier because if Docker is installed, you simply do a docker build, then you can run the container directly from the local cache when running the vendoring script. What do you think of this idea instead- the locally built container can be vetted easily as well, I feel?

[Lukasa]

6.x is totally fine. And I do quite like the idea of using the custom SDK support, in part because it will be a lot faster: cross-compiling will work substantially better than qemu-based emulation.

[xtremekforever]

@Lukasa of course, cross compilation is the way 💪 however I will note that the Docker solution also cross compiles (and runs Debian/Ubuntu natively in the container) but using the older destination.json SDK installed at /opt. The container is based on swift:<version> with just the SDK slapped on top.

So, it may also be another solution as well- but I'll see what you think about the Swift SDK generator first so you can get a sense for it.