Add AOT building #147

provokateurin · 2020-06-01T21:35:08Z

Needs https://github.com/go-flutter-desktop/go-flutter/tree/aot, because the file names that go-flutter links against had to be moved to hover (hover build linux -b @aot).
Hover now defaults to AOT builds when using hover build ...
Hover restricts the user to building AOT apps only on the native platform (cross-compiling windows AOT on linux using wine and darwin AOT on linux using darling already works, but we need to think about how exactly to implement that feature)

Closes go-flutter-desktop/go-flutter#259
Closes go-flutter-desktop/go-flutter#452

pchampio · 2020-06-02T09:01:14Z

Awesome work!!!
The share lib that you are linking go-flutter against contains:

C function ABI needed for go-flutter
The flutter provided shell (flutter alternative to go-flutter)

I don't think it matter that much, the shell lib seems to only be slightly bigger than the 'raw' engine embedding ABI lib.

The issues I see:

On linux, the GLFW external lib is used as the ABI lib. At some point, we might propose other render-er toolkit than GLFW. In the future, or, we might support non-X11 display like what's available in https://github.com/ardera/flutter-pi.
At the moment, go-flutter bundles the GLFW lib into the app. My issue is as followed, If go-flutter don't comes with the GLFW bindings, if the systems doesn't have GLFW bindings, does the downloaded shell lib still work? (on windows and macos, I don't think it will cause any issues because they are using native render-er toolkit)
The flutter team is planning to stop using the GLFW shell and use an GTK one. Will GLFW shell builds always be available, they might stop pushing them to their infra.. If we had to use the GTK shell instead, of the GLFW one, my above question also applies.

The usage of the shell lib instead of embedding ABI is LGTM for me. go-flutter is capable of AOT and it will allow more people to test AOT.
Still, I think the best way to link go-flutter is with the raw ABI and not the shell impl. This is blocked by: flutter/flutter#44577
We know https://github.com/flutter-rs/engine-builds is an alternative, but I don't like hand crafted builds that fails and requires manual maintenance.

etx · 2020-06-02T14:35:28Z

Testing this out based on @jld3103 request.

MacOS 10.15.5
Flutter master channel
Go 1.14.1

Build:
$hover build darwin -b @test/remove-cgo-ldflags hover: Using engine from cache hover: Cleaning the build directory hover: ⚠ The go-flutter project tries to stay compatible with the beta channel of Flutter. hover: ⚠ It's advised to use the beta channel: flutter channel beta hover: Bundling flutter app Font subetting is not supported in debug mode. The --tree-shake-icons flag will be ignored. hover: Generating kernel snapshot hover: Generating ELF snapshot Warning: Generating ELF library without DWARF debugging information. hover: Downloading 'go-flutter' @test/remove-cgo-ldflags go: golang.org/x/text upgrade => v0.3.2 hover: 'go-flutter' is on version: v0.41.1-0.20200601082658-85c2555a90e0 hover: Compiling 'go-flutter' and plugins hover: Successfully compiled executable binary for darwin

When I run the output bin:

go-flutter: WARNING error creating the resource window: VersionUnavailable: NSGL: Failed to create OpenGL context
[FATAL:flutter/runtime/dart_vm.cc(420)] Error while initializing the Dart VM: Flag dedup_instructions is false in snapshot, but dedup_instructions is always true in product mode
SIGABRT: abort
PC=0x7fff695fe33a m=0 sigcode=0
signal arrived during cgo execution

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x43b3740, 0xc00013f878, 0xc000142380)
	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/cgocall.go:133 +0x5b fp=0xc00013f848 sp=0xc00013f810 pc=0x400597b
github.com/go-flutter-desktop/go-flutter/embedder._Cfunc_runFlutter(0xc000027048, 0xc0001270e0, 0xc000142380, 0x0)
	_cgo_gotypes.go:853 +0x4d fp=0xc00013f878 sp=0xc00013f848 pc=0x432739d
github.com/go-flutter-desktop/go-flutter/embedder.(*FlutterEngine).Run.func7(0xc000027048, 0xc0001270e0, 0xc000142380, 0xc00013f908)
	/Users/dkuschel/go/pkg/mod/github.com/go-flutter-desktop/go-flutter@v0.41.1-0.20200601082658-85c2555a90e0/embedder/embedder.go:150 +0xe4 fp=0xc00013f8a8 sp=0xc00013f878 pc=0x4329dd4
github.com/go-flutter-desktop/go-flutter/embedder.(*FlutterEngine).Run(0xc0001270e0, 0xc000027048, 0xc000115f20, 0x3, 0x3, 0x0, 0x0)
	/Users/dkuschel/go/pkg/mod/github.com/go-flutter-desktop/go-flutter@v0.41.1-0.20200601082658-85c2555a90e0/embedder/embedder.go:150 +0x337 fp=0xc00013fb48 sp=0xc00013f8a8 pc=0x4328437
github.com/go-flutter-desktop/go-flutter.(*Application).Run(0xc0001422a0, 0x0, 0x0)
	/Users/dkuschel/go/pkg/mod/github.com/go-flutter-desktop/go-flutter@v0.41.1-0.20200601082658-85c2555a90e0/application.go:250 +0x915 fp=0xc00013fec0 sp=0xc00013fb48 pc=0x434c275
github.com/go-flutter-desktop/go-flutter.Run(0xc00012e5a0, 0xb, 0x12, 0x9, 0xc00013ff68)
	/Users/dkuschel/go/pkg/mod/github.com/go-flutter-desktop/go-flutter@v0.41.1-0.20200601082658-85c2555a90e0/application.go:26 +0x4d fp=0xc00013fef0 sp=0xc00013fec0 pc=0x434affd
main.main()
	/Users/dkuschel/Desktop/Repos/thanos/go/cmd/main.go:24 +0x188 fp=0xc00013ff88 sp=0xc00013fef0 pc=0x438be78
runtime.main()
	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/proc.go:203 +0x212 fp=0xc00013ffe0 sp=0xc00013ff88 pc=0x40370a2
runtime.goexit()
	/usr/local/Cellar/go/1.14.1/libexec/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00013ffe8 sp=0xc00013ffe0 pc=0x4063a51

rax    0x0
rbx    0x780edc0
rcx    0x7ffeefbfca48
rdx    0x0
rdi    0x307
rsi    0x6
rbp    0x7ffeefbfca70
rsp    0x7ffeefbfca48
r8     0x130a8
r9     0x0
r10    0x780edc0
r11    0x246
r12    0x307
r13    0x53792b0
r14    0x6
r15    0x16
rip    0x7fff695fe33a
rflags 0x246
cs     0x7
fs     0x0
gs     0x0

provokateurin · 2020-06-02T14:39:49Z

Ok thanks for trying! I think it's fairly easy to fix it

pchampio · 2020-06-03T07:38:26Z

There is this issue to keep in mind: flutter/flutter#38935

https://flutter-review.googlesource.com/c/recipes/+/3220/2/recipes/engine.py#1330
Example:

- flutter_infra/flutter/2d72510e447ab60a9728aeea2362d8be2cbd7789/linux-x64/linux-x64-flutter-gtk.zip
+ flutter_infra/flutter/2d72510e447ab60a9728aeea2362d8be2cbd7789/linux-x64-debug/linux-x64-flutter-gtk.zip

https://flutter-review.googlesource.com/c/recipes/+/3220

pchampio · 2020-06-04T09:53:52Z

This doesn't work anymore: flutter/engine#18735

provokateurin · 2020-06-04T14:13:23Z

The problem with MacOS seems that it uses assembly code and not a shared ELF library (See https://github.com/flutter/flutter/blob/cd593dae1961ca2bc5c045a310f16e6cdffc8294/packages/flutter_tools/lib/src/base/build.dart#L146).
@pchampio have you and clue on how to implement AOT from assembly in go-flutter?

pchampio · 2020-06-04T14:19:50Z

The problem with MacOS seems that it uses assembly code and not a shared ELF library (See https://github.com/flutter/flutter/blob/cd593dae1961ca2bc5c045a310f16e6cdffc8294/packages/flutter_tools/lib/src/base/build.dart#L146).

The flutter team didn't had flutter/flutter#50778 to read shared ELF library in a cross platform manner when they first implemented AOT on macos. Now that flutter/flutter#50778 is available, we should provide an ELF for all platforms.

I don't want to go through --snapshot_kind=app-aot-assembly because it requires x-code to later compile the app.

provokateurin · 2020-06-04T14:28:44Z

Hm I'm not sure if AOT on MacOS is then going to work for now. The current error I'm stuck on is this:

[FATAL:flutter/runtime/dart_vm.cc(420)] Error while initializing the Dart VM: Snapshot not compatible with the current VM configuration: the snapshot requires 'release no-dwarf_stack_traces_mode no-causal_async_stacks lazy_async_stacks use_bare_instructions dedup_instructions no-"asserts" no-code-comments x64-sysv no-null-safety' but the VM has 'product no-dwarf_stack_traces_mode no-causal_async_stacks lazy_async_stacks use_bare_instructions dedup_instructions no-"asserts" x64-sysv no-null-safety'

The --no-code-comments flag doesn't seem to work and product should be release, but I don't know how to change it.

provokateurin · 2020-06-05T14:07:12Z

This doesn't work anymore: flutter/engine#18735

What can we do about it?

pchampio · 2020-06-05T14:10:00Z

Wait for: flutter/flutter#44577
;(

provokateurin · 2020-06-05T14:19:33Z

Ouch. AOT was working a few days and now it's gone again... sad :(

stuartmorgan-g · 2020-06-06T20:05:03Z

Ouch. AOT was working a few days and now it's gone again... sad :(

I was just pointed to this PR; I'm sorry that the bug wasn't fixed much earlier so that it would have been clear up front that this wasn't a viable approach.

provokateurin · 2020-08-30T09:05:17Z

https://github.com/flutter-rs/engine-builds seems to be working properly again. I'm going to change the engine downloads URLs so this PR will work again and then we can see how far everything works.

pchampio · 2020-08-30T12:49:07Z

@jld3103 Yea, good idea!
I think project like https://github.com/flutter-rs/engine-builds are unstable when compiling against the flutter master channel.
Regarding the beta channel, I think/hope it's quite stable.

provokateurin · 2020-08-30T20:58:02Z

I needed to align some flag usage between build and run and also simplified a lot of other stuff. It's working on linux and on windows through although somehow the font was missing (app comes with font bundled). I need to check if the lost font is due to my changes or also happened before. I will push the changes tomorrow to test them in my MacOS VM and for others to test. Getting it to work on MacOS seems to have gotten easier, because the darwin engine uses a shared library like linux and windows and not a framework like in the official engine builds. I have high hopes that it will work.

provokateurin · 2020-08-30T21:00:25Z

@pchampio Btw is there anything holding us back from merging the aot branch into the master branch of go-flutter?

pchampio · 2020-08-31T15:01:41Z

@pchampio Btw is there anything holding us back from merging the aot branch into the master branch of go-flutter?

No, the aot branch can be merged on the go-flutter side!

provokateurin · 2020-08-31T15:23:48Z

It seems to be working on MacOS! I need to make some small changes and then it works completely. Someone would need to test it on a real machine, but I'm 99% sure it will work there.

provokateurin · 2020-08-31T15:27:10Z

@pchampio Btw is there anything holding us back from merging the aot branch into the master branch of go-flutter?

No, the aot branch can be merged on the go-flutter side!

I need to change some things in go-flutter and I will put them directly into the aot branch so please don't merge yet.

provokateurin · 2020-09-01T12:45:43Z

I got everything to work! I needed to create a hack to fix the linking of the flutter engine to the executable, because it was using a relative path which wouldn't work if you weren't in the directory of the executable. The changes in go-flutter needed for this to work are now in the aot branch so simply use hover build ... -b @aot.
While making these changes I used wine and darling for building so in the future it will be possible to cross-compile an AOT app, but that is something to worry about later.

@etx would you be able to test this on a real Mac? That would help a lot

GeertJohan · 2020-09-12T10:35:20Z

internal/enginecache/cache.go

 	if err != nil {
-		log.Warnf("%v", err)
+		log.Errorf("Failed to download engine: %v", err)
+		log.Infof("That may mean no engine download is currently available. You'll have to wait for one to get available")


Maybe we can improve the wording a bit here and give better advice.

A possible "solution" for the user may be to go back an older flutter version. We could advice the user to use the beta channel if they aren't using that already, since those engines are more likely to be available.

We may also explain where we get the engine builds from, so the user knows where to look which versions are available.

Looking up the versions manually isn't really fun. We could just download the engine versions for all branches and then tell the user which ones are already built so they can use that branch.

I've changed the wording, but it's still not perfect

provokateurin · 2020-09-12T16:55:24Z

Hover now requires v0.42.0 of go-flutter and upgrades the project if the version requirement isn't met. Everything else is backwards compatible.

cmd/build.go

provokateurin force-pushed the feature/aot branch from 0ddcac2 to dc8eb8f Compare June 2, 2020 06:19

provokateurin force-pushed the feature/aot branch 2 times, most recently from 4f2e8e8 to 2ca674f Compare June 2, 2020 20:13

pchampio mentioned this pull request Jun 4, 2020

Garbled text when external texture is used go-flutter-desktop/go-flutter#463

Open

provokateurin force-pushed the feature/aot branch from 2ca674f to 861c9b1 Compare June 4, 2020 12:43

ohir mentioned this pull request Jun 6, 2020

Don't export embedder API in desktop embeddings flutter/engine#18735

Merged

provokateurin marked this pull request as draft June 25, 2020 05:19

provokateurin mentioned this pull request Jul 22, 2020

What's the status on AOT? go-flutter-desktop/go-flutter#496

Closed

provokateurin force-pushed the feature/aot branch from 861c9b1 to 2465697 Compare August 30, 2020 09:11

provokateurin force-pushed the feature/aot branch from 2465697 to 9c2a94e Compare September 1, 2020 12:38

provokateurin marked this pull request as ready for review September 1, 2020 12:45

provokateurin force-pushed the feature/aot branch 3 times, most recently from 260d4c6 to 87b219f Compare September 2, 2020 17:14

GeertJohan reviewed Sep 12, 2020

View reviewed changes

provokateurin force-pushed the feature/aot branch from 87b219f to 17faead Compare September 12, 2020 16:53

Add AOT building

9ad26f5

provokateurin force-pushed the feature/aot branch from 17faead to 9ad26f5 Compare September 12, 2020 16:58

GeertJohan requested changes Oct 12, 2020

View reviewed changes

cmd/build.go Show resolved Hide resolved

GeertJohan approved these changes Oct 12, 2020

View reviewed changes

provokateurin merged commit 1e7668b into master Oct 12, 2020

provokateurin deleted the feature/aot branch October 12, 2020 09:37

provokateurin mentioned this pull request Oct 26, 2020

Add automatic and manual cache invalidation #172

Merged

Add AOT building #147

Add AOT building #147

Uh oh!

Conversation

provokateurin commented Jun 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pchampio commented Jun 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

etx commented Jun 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

provokateurin commented Jun 2, 2020

Uh oh!

pchampio commented Jun 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pchampio commented Jun 4, 2020

Uh oh!

provokateurin commented Jun 4, 2020

Uh oh!

pchampio commented Jun 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

provokateurin commented Jun 4, 2020

Uh oh!

provokateurin commented Jun 5, 2020

Uh oh!

pchampio commented Jun 5, 2020

Uh oh!

provokateurin commented Jun 5, 2020

Uh oh!

stuartmorgan-g commented Jun 6, 2020

Uh oh!

provokateurin commented Aug 30, 2020

Uh oh!

pchampio commented Aug 30, 2020

Uh oh!

provokateurin commented Aug 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

provokateurin commented Aug 30, 2020

Uh oh!

pchampio commented Aug 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

provokateurin commented Aug 31, 2020

Uh oh!

provokateurin commented Aug 31, 2020

Uh oh!

provokateurin commented Sep 1, 2020

Uh oh!

GeertJohan Sep 12, 2020

Choose a reason for hiding this comment

Uh oh!

provokateurin Sep 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

provokateurin Sep 12, 2020

Choose a reason for hiding this comment

Uh oh!

provokateurin commented Sep 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

provokateurin commented Jun 1, 2020 •

edited

Loading

pchampio commented Jun 2, 2020 •

edited

Loading

etx commented Jun 2, 2020 •

edited

Loading

pchampio commented Jun 3, 2020 •

edited

Loading

pchampio commented Jun 4, 2020 •

edited

Loading

provokateurin commented Aug 30, 2020 •

edited

Loading

pchampio commented Aug 31, 2020 •

edited

Loading

provokateurin Sep 12, 2020 •

edited

Loading

provokateurin commented Sep 12, 2020 •

edited

Loading