Envoy on Apple silicon m1 #16482
Labels
Comments
JCzz
added
enhancement
|
cc @ggreenway who may know more about any toolchain issues preventing build for m1's |
|
Istio / arm is not supported Envoy not supporting arm? |
|
It's on my TODO list to investigate building envoy for m1, but I don't know when I'll get to it. If someone else wants to take it up, go for it. |
|
One of the challenges is availability of VMs with m1 silicon or physical hardware to test on. |
|
I've heard that we could get a single m1 instance for the OSS project from https://www.macstadium.com/opensource. It might be enough to setup a periodic build off of main to post-facto check for breakages. Also on my TODO list to look into. |
|
My m1 Mac Mini arrived last night (16GB RAM) and I've got the development environment set up, full xcode installed, the brand new m1 native I'm getting the following error on the first thing the process tries to compile. The Now wondering if I should try getting the m1 compatible Azul JDK and building Bazel from source? |
|
I now have a working binary on Apple Silicon. My goal was to not have to install Rosetta:
This compilation worked and produced a binary that wouldn't run generating a That lead me to - wg/wrk#448 bazel/envoy_binary.bzl | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/bazel/envoy_binary.bzl b/bazel/envoy_binary.bzl
index 43c52bf7e..3bd40e35d 100644
--- a/bazel/envoy_binary.bzl
+++ b/bazel/envoy_binary.bzl
@@ -57,8 +57,8 @@ def _envoy_linkopts():
# The macOS system library transitively links common libraries (e.g., pthread).
"@envoy//bazel:apple": [
# See note here: https://luajit.org/install.html
- "-pagezero_size 10000",
- "-image_base 100000000",
+ # "-pagezero_size 10000",
+ # "-image_base 100000000",
],
"@envoy//bazel:windows_opt_build": [
"-DEFAULTLIB:ws2_32.lib", This will need to be updated with correct logic.
bazel/BUILD | 1 +
1 file changed, 1 insertion(+)
diff --git a/bazel/BUILD b/bazel/BUILD
index 98be980f8..66f2e364d 100644
--- a/bazel/BUILD
+++ b/bazel/BUILD
@@ -540,6 +540,7 @@ selects.config_setting_group(
name = "apple",
match_any = [
":darwin",
+ ":darwin_arm64",
":darwin_x86_64",
":ios_arm64",
":ios_arm64e", Success! |
|
Hi @moderation that is super cool work! |
|
Hi @ggreenway did you see @moderation comments? |
|
Hi, I tried building the latest codebase using the patch suggested above. Although that did get me further than before I'm now seeing the following error on M1 I'm using the latest bazel to build envoy and ran |
|
Also, if anyone is able to build envoy on arm64 (as suggested on this issue), can you please share the binary so others can try it out? |
|
@vinayreddy are you talking about M1 arm64 specifically or just arm64? There are arm64 images at https://hub.docker.com/r/envoyproxy/envoy/tags?page=1&ordering=last_updated. This is Ok if you have Docker. There are some hacky script ways to extract the binary from the images I'm building on M1 but I compile out a bunch of extensions and you probably shouldn't trust a binary from random people on the internet 😀 |
|
Yes, I was talking M1 arm64. Yes, I was able to run a docker image on my M1 mac, so that is something I can use to make progress. Haha, fair point that I shouldn't run random binaries I find on the Internet. |
|
@vinayreddy good that you have an image running on your M1. I think today the only way to get an M1 Apple Silicon binary, that can run without Rosetta or virtualization like Docker, is the process I outlined at #16482 (comment) |
This was referenced Jul 8, 2021
|
Does someone have resources and build skills to implement what @moderation said above? There's considerable tech debt relying on translation as for example you have to consider the intent when this is eventually addressed. The longer this takes to resolve the more ecosystem cleanup will be needed. |
|
@JCzz did you close this by accident? it isn't obvious what fixed it.. |
|
@keith just as a try. this isn't resolved and one issue we have in homebrew is that few of the envoy dimensions work. Notably, many have apple silicon at this point and while possible to use rossetta 2, it seems like if some effort started on arm64 support for mac, it might finish, but if nothing starts and this stays closed it will never resolve. |
|
I'm not really sure what is required here but I will mark this help wanted. |
mattklein123
added
area/macos
and removed
enhancement
keith
added a commit
to keith/envoy
that referenced
this issue
Nov 6, 2021
I'm not sure what changed here but it seems chromium attempts to use lld on macOS even though it doesn't ship with Xcode. This could be intentional on their part if they require folks to use an llvm toolchain instead, but this likely isn't the right move for envoy unless we vendor that toolchain ourselves. For now we can safely disable this. envoyproxy#16482 (comment) Signed-off-by: Keith Smiley <keithbsmiley@gmail.com>
|
Fix here #18922 |
lizan
pushed a commit
that referenced
this issue
Nov 7, 2021
I'm not sure what changed here but it seems chromium attempts to use lld on macOS even though it doesn't ship with Xcode. This could be intentional on their part if they require folks to use an llvm toolchain instead, but this likely isn't the right move for envoy unless we vendor that toolchain ourselves. For now we can safely disable this. #16482 (comment) Signed-off-by: Keith Smiley <keithbsmiley@gmail.com>
|
Here's another improvement for everyone, but previously was only blocked by M1 support #19072 |
|
Ok the long running rules_go / golang update landed #17460 so we'll have to test end to end again but I think the previously known blockers are all resolved! |
|
Alright I just tested a |
|
@keith should we update rules_go from 0.27.0 to 0.29.0 to unblock M1 build support? |
|
We're on 0.28 now, it landed as part of that PR above envoy/bazel/repository_locations.bzl Line 654 in 70a5f29 I think we're unblocked here, post here if you see issues |
|
thanks for the progress @keith. The goal isn't so much me working, but homebrew working (#17500). I won't have time to resolve all the issues, but I can share where things are right now. Firstly, while envoy's build requires git (#2181), you need a git tag in order to test whether homebrew works or not. (cc @carlocab in case I am mistaken) Anyway, as that seems to be the case, and I don't have karma to push a tag, I made a fork and pushed a tag for the latest commit, then updated my local copy of envoy.rb to point to that. Ex. this is what to change $ cd /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/
$ git diff Formula/envoy.rb
diff --git a/Formula/envoy.rb b/Formula/envoy.rb
index ce4a24d6d6a..94760519a70 100644
--- a/Formula/envoy.rb
+++ b/Formula/envoy.rb
@@ -3,9 +3,9 @@ class Envoy < Formula
homepage "https://www.envoyproxy.io/index.html"
# Switch to a tarball when the following issue is resolved:
# https://github.com/envoyproxy/envoy/issues/2181
- url "https://github.com/envoyproxy/envoy.git",
- tag: "v1.20.1",
- revision: "ea23f47b27464794980c05ab290a3b73d801405e"
+ url "https://github.com/codefromthecrypt/envoy.git",
+ tag: "v1.21.0",
+ revision: "8a9779a615afebbff8b765d3fdd99c8c0c818591"
license "Apache-2.0"
# Apple M1/arm64 is pending envoyproxy/envoy#16482Next, try $ brew install --build-from-source envoy
==> Downloading https://ghcr.io/v2/homebrew/core/cmake/manifests/3.22.0
Already downloaded: /Users/thebots/Library/Caches/Homebrew/downloads/0e6c1beadc6246cd1a5760e035b71640ac5673ffbee411d0c696f7c4a40aae96--cmake-3.22.0.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/cmake/blobs/sha256:2ecfdea833356807af17016cd40db4d51f04ad6c00479e
Already downloaded: /Users/thebots/Library/Caches/Homebrew/downloads/ac6eeaf46044bb24578718e225345f6294fb35fb3f2faa73f188a6d4d736cb92--cmake--3.22.0.arm64_monterey.bottle.tar.gz
==> Downloading https://ghcr.io/v2/homebrew/core/ninja/manifests/1.10.2_1
Already downloaded: /Users/thebots/Library/Caches/Homebrew/downloads/3d885b06ee8943649e1e4f3edebcca2003dc2e9e332202cc2a0b16ccce4f8cae--ninja-1.10.2_1.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/ninja/blobs/sha256:8efac5a9c8b7028f64f5a092eb029ff40887b9895fe423
Already downloaded: /Users/thebots/Library/Caches/Homebrew/downloads/deb592b6fb7dd715b57a170608e055406d978bbc68c284ffa7ecef67acef899a--ninja--1.10.2_1.arm64_monterey.bottle.tar.gz
==> Cloning https://github.com/codefromthecrypt/envoy.git
Updating /Users/thebots/Library/Caches/Homebrew/envoy--git
From https://github.com/codefromthecrypt/envoy
* [new tag] v1.21.0 -> v1.21.0
==> Checking out tag v1.21.0
Previous HEAD position was 96701cb24 docs: fixes for release notes for 1.20 (#18403)
HEAD is now at 8a9779a61 build(deps): bump setuptools from 59.0.1 to 59.5.0 in /tools/base (#19201)
HEAD is now at 8a9779a61 build(deps): bump setuptools from 59.0.1 to 59.5.0 in /tools/base (#19201)
Error: v1.21.0 tag should be 9d5627a0879b0a029e90515137c108e1d2884bfc
but is actually 8a9779a615afebbff8b765d3fdd99c8c0c818591
MacBook-Air:homebrew thebots$ vi ./Library/Taps/homebrew/homebrew-core/Formula/envoy.rb
MacBook-Air:homebrew thebots$ brew install --build-from-source envoy
==> Downloading https://ghcr.io/v2/homebrew/core/cmake/manifests/3.22.0
Already downloaded: /Users/thebots/Library/Caches/Homebrew/downloads/0e6c1beadc6246cd1a5760e035b71640ac5673ffbee411d0c696f7c4a40aae96--cmake-3.22.0.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/cmake/blobs/sha256:2ecfdea833356807af17016cd40db4d51f04ad6c00479e
Already downloaded: /Users/thebots/Library/Caches/Homebrew/downloads/ac6eeaf46044bb24578718e225345f6294fb35fb3f2faa73f188a6d4d736cb92--cmake--3.22.0.arm64_monterey.bottle.tar.gz
==> Downloading https://ghcr.io/v2/homebrew/core/ninja/manifests/1.10.2_1
Already downloaded: /Users/thebots/Library/Caches/Homebrew/downloads/3d885b06ee8943649e1e4f3edebcca2003dc2e9e332202cc2a0b16ccce4f8cae--ninja-1.10.2_1.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/ninja/blobs/sha256:8efac5a9c8b7028f64f5a092eb029ff40887b9895fe423
Already downloaded: /Users/thebots/Library/Caches/Homebrew/downloads/deb592b6fb7dd715b57a170608e055406d978bbc68c284ffa7ecef67acef899a--ninja--1.10.2_1.arm64_monterey.bottle.tar.gz
==> Cloning https://github.com/codefromthecrypt/envoy.git
Updating /Users/thebots/Library/Caches/Homebrew/envoy--git
==> Checking out tag v1.21.0
HEAD is now at 8a9779a61 build(deps): bump setuptools from 59.0.1 to 59.5.0 in /tools/base (#19201)
HEAD is now at 8a9779a61 build(deps): bump setuptools from 59.0.1 to 59.5.0 in /tools/base (#19201)
==> Installing dependencies for envoy: cmake and ninja
==> Installing envoy dependency: cmake
==> Pouring cmake--3.22.0.arm64_monterey.bottle.tar.gz
🍺 /opt/homebrew/Cellar/cmake/3.22.0: 2,986 files, 41.6MB
==> Installing envoy dependency: ninja
==> Pouring ninja--1.10.2_1.arm64_monterey.bottle.tar.gz
🍺 /opt/homebrew/Cellar/ninja/1.10.2_1: 10 files, 388KB
==> Installing envoy
==> /opt/homebrew/opt/bazelisk/bin/bazelisk build --compilation_mode=opt --curses=no --show_task_finish --verbose_
Last 15 lines from /Users/thebots/Library/Logs/Homebrew/envoy/01.bazelisk:
Analyzing: target //source/exe:envoy-static (20 packages loaded, 7 targets configured)
Analyzing: target //source/exe:envoy-static (20 packages loaded, 7 targets configured)
Analyzing: target //source/exe:envoy-static (20 packages loaded, 7 targets configured)
Analyzing: target //source/exe:envoy-static (20 packages loaded, 7 targets configured)
Analyzing: target //source/exe:envoy-static (20 packages loaded, 7 targets configured)
Analyzing: target //source/exe:envoy-static (20 packages loaded, 7 targets configured)
ERROR: /private/tmp/envoy-20211207-61335-aduab4/.brew_home/_bazel/dd0c3ef926b0a56f33d0ac8f5e25816a/external/local_config_cc/BUILD:48:19: in cc_toolchain_suite rule @local_config_cc//:toolchain: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'darwin_arm64'
ERROR: While resolving configuration keys for //bazel:gcc_build: Analysis of target '@local_config_cc//:toolchain' failed
ERROR: While resolving configuration keys for //source/exe:envoy-static: Analysis of target '@local_config_cc//:toolchain' failed
ERROR: Analysis of target '//source/exe:envoy-static' failed; build aborted: Analysis of target '@local_config_cc//:toolchain' failed
INFO: Elapsed time: 59.529s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (34 packages loaded, 202 targets configured)
FAILED: Build did NOT complete successfully (34 packages loaded, 202 targets configured)
READ THIS: https://docs.brew.sh/Troubleshooting |
|
Thanks for testing, I've submitted Homebrew/homebrew-core#90601 which allows you to do Is your host an M1 mac? Do you have a full installation of Xcode vs only the command line tools? |
|
I tested on my m1 mac. |
|
@keith I fixed the above problem by |
|
@carlocab (but anyone can answer) do you know if there's a way to avoid
explicit settings like above? Seems we want to avoid configuration, ideally
zero.
…On Tue, Dec 7, 2021 at 4:01 PM 白泽 ***@***.***> wrote:
@keith <https://github.com/keith> I fixed the above problem by sudo
xcode-select --switch /Applications/Xcode.app. And It has built
successfully. GOOD JOB!! THKS!!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#16482 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAPVV5MIBYY3PPQYNXGVWDUPW5PJANCNFSM442RLFUQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
|
@codefromthecrypt The need to install XCODE and configure it correctly is probably unavoidable. Fortunately, the configuration is correct after installed XCODE in normal. |
|
@Patrick0308 would you happen to know what Does doing sudo xcode-select --switch /Library/Developer/CommandLineToolsnot work? |
|
@carlocab Sorry, I didn't print the PATH of the active developer directory before did |
|
Ah. Chromium. The formula probably needs a depends_on xcode: :build |
|
ok trying it on both my Macs prior to marking ready for review. 🤞 and thanks folks. seems we're nearly there (pending an envoy release) Homebrew/homebrew-core#90695 |
|
The error is now improved, if full Xcode isn't installed. Thanks for figuring this out @Patrick0308 and thanks for the config tip @carlocab! |
|
I almost did nothing except |
|
After installing, switching to, and accepting the license of full Xcode, the current main branch works for me on my M1 laptop! Looking forward to 1.21 as it will allow homebrew to remove the arm64 exception! |
|
I build successfully now. I'm not quite sure what the problem was that caused built failed. I just uninstalled anaconda and it works. I think this issue can be closed. |
|
In general I think we should close this one and encourage folks to file new issues as they pop up |
|
hey folks, is m1 fast at compiling envoy? Do people have a good dev experience with m1? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Title: Envoy on Apple silicon m1
Hi
I am trying to get Knative with Istio(or any other Ingress controller based on Envoy) running on a Mac mini with the Apple Silicon m1 chip. And I am told that Istio is not working do to Envoy not supporting Apple m1 chip.
Description:
A working setup on Kubernetes running Istio(Envoy)
The text was updated successfully, but these errors were encountered: