fix: shorebird engine builds are larger than expected, fix and document size

[bryanoltman]

Description

Comparing the output of flutter build apk --release vs. shorebird build apk --release should give you a rough idea.

[pushpendraKh]

I did it on default Flutter's counter project with and without Shorebird.

Without the Shorebird, the size was 17.9, and with the shorebird size was 24.5. Diff is ~7MB which is more than what is mentioned in the document (1MB). // @eseidel

[eseidel]

Will investigate, thanks.

[eseidel]

Yeah, something is very wrong. I wrote a tool to compare:

eseidel@erics-mbp compare_sizes % dart run
Building package executable... 
Built compare_sizes:compare_sizes.
shorebird_engine_sha not specified, defaulting to https://github.com/shorebirdtech/flutter/shorebird/dev/bin/internal/engine.version
Shorebird engine: e7bdf38443dff65d64446d6ca4074a35a587c770
Flutter engine:   1ac611c64eadbd93c5f5aba5494b8fc3b35ee952
Shorebird: 17.57 MB
Flutter:   9.33 MB

[eseidel]

I added support for Android, it's also bigger than expected.

These iOS sizes are uncompressed (which isn't what will ship) and the Android sizes are semi-compressed. The jar itself is compressed, but will be re-compressed as part of the apk.

eseidel@erics-mbp compare_sizes % dart run
Building package executable... 
Built compare_sizes:compare_sizes.
shorebird_engine_sha not specified, defaulting to https://github.com/shorebirdtech/flutter/shorebird/dev/bin/internal/engine.version
Shorebird engine: e7bdf38443dff65d64446d6ca4074a35a587c770
Flutter engine:   1ac611c64eadbd93c5f5aba5494b8fc3b35ee952
ios-release, Flutter.xcframework/ios-arm64/Flutter.framework/Flutter:
  Shorebird 17.57 MB
  Flutter   9.33 MB
android-arm64-release, flutter.jar:
  Shorebird 7.36 MB
  Flutter   5.21 MB

[eseidel]

Comparing https://github.com/shorebirdtech/build_engine/blob/main/build_engine/mac_build.sh and https://github.com/flutter/engine/blob/main/ci/builders/mac_ios_engine.json I don't see any obvious compile-flag differences.

[eseidel]

This could explain it: shorebirdtech/buildroot@d6c410f If there are a ton of symbols that slip through on iOS as a result?

[eseidel]

https://github.com/evmar/bloat is a tool we used to use for this on Chromium.

[eseidel]

I suspect this could be part of our problem:
rust-lang/cargo#4122

I'm not actually sure how that interacts with .a files (if it does at all), since I would expect the stripping to happen at a later stage?

[eseidel]

I've compared the nm output of Flutter vs. Shorebird on iOS and found that Shorebird has tons more symbols (most/all rust?) likely due to:
shorebirdtech/buildroot@57142b8

Still trying to figure out what the right way to expose only the symbols we need from our Rust code into the Flutter Engine w/o causing (at least iOS) to also then re-expose all of those symbols in Flutter.framework.

[eseidel]

OK, learned a bit about stripping our rust output tonight. This should work:

ld -r -o foo.o -exported_symbols_list library/symbols.exports target/aarch64-apple-ios/release/libupdater.a -arch arm64 -x

-x makes it hide/mangle all non-global names so they dead strip correctly.

With this exports list:

_shorebird_check_for_update
_shorebird_current_boot_patch_number

(Need to add the rest of those to make this real.)

We may need to turn the .o back into a .a to link it into libflutter, using ar, can try that tomorrow.

[eseidel]

     -r      Merges object files to produce another mach-o object file with file type MH_OBJECT.
     -x      Do not put non-global symbols in the output file's symbol table. Non-global symbols are useful when debugging and getting symbol names in back traces, but are not used at runtime. If -x is
             used with -r non-global symbol names are not removed, but instead replaced with a unique, dummy name that will be automatically removed when linked into a final linked image.  This allows dead
             code stripping, which uses symbols to break up code and data, to work properly and provides the security of having source symbol names removed.

[eseidel]

Generated symbols.exports with:

objdump target/aarch64-apple-ios/release/libupdater.a --syms | grep shorebird | awk '{ print $5}' > symbols.exports 

[eseidel]

I added this into our rust build script, only to then realize that rust build scripts run before cargo build, not after, so this can't work:

   // We only do our re-link for iOS targets right now.
    let target = env::var("TARGET").unwrap();
    if target != "aarch64-apple-ios" {
        return;
    }

    // ld -r -o foo.o -exported_symbols_list library/symbols.exports target/aarch64-apple-ios/release/libupdater.a -arch arm64 -x
    let out_dir = env::var("OUT_DIR").unwrap();
    // list files in outdir
    let crate_root = env::var("CARGO_MANIFEST_DIR").unwrap();
    let symbols_path = Path::new(&crate_root).join("symbols.exports");
    let status = Command::new("ld")
        .args(&[
            // -r Merges object files to produce another mach-o object file with file type MH_OBJECT.
            "-r",
            // -x Do not put non-global symbols in the output file's symbol table.
            // Hides non-global symbols from the output file's symbol table and
            // makes dead code stripping work.
            "-x",
            // arch is required when working with mach-o due to fat binaries.
            "-arch",
            "arm64",
            "-exported_symbols_list",
            symbols_path.to_str().unwrap(),
            "-o",
            "libupdater.o",
            "libupdater.a",
        ])
        .current_dir(&Path::new(&out_dir))
        .status()
        .unwrap();
    panic!("status: {}", status);

Will work on a change to our gn/ninja files instead.

[eseidel]

https://darkcoding.net/software/a-very-small-rust-binary-indeed/ may also be helpful.

[eseidel]

Thought about this more. The steps to fix this:

1. Set up tracking of our binary size for an application over time.
2. That will probably reveal that our rust code is adding all the size. The fix for that is probably to translate it to C++ code, although it's possible that we can simply make the Rust code smaller by removing most of its dependencies and instead calling back to the C++ code through C functions to do the yaml parse, http loads, etc.

[eseidel]

When we get to this shorebirdtech/engine#49 was one start, but we should measure first.