"Can't reconcile two non-macho files" after upgrading to 24.x.x

[Nantris]

• Electron-Builder Version: 24.1.1

• Node Version: 16.17.0

• Electron Version: 23.3.x
• Electron Type (current, beta, nightly):

Current

• Target: mac-universal

It's presumably related to this underlying issue: electron/universal#41

I know there's already 3 other issues about this, but they all seem stale and different in their cause.

[Nantris]

We also see this error instead intermittently: Detected unique file "node_modules/argon2/bin/darwin-arm64-113" in "/Users/me/Documents/app/release/prod/mac-universal--arm64/app.app/Contents/Resources/app.asar" not covered by allowList rule: "undefined"

But... the file that is causing the problem always seems to be different, whichever error type we get.

[Nantris]

Copying my comment from electron/universal#41:

---

In my case adding '!node_modules/**/*.{mk,a,o,h}' just results in it failing on some other file, and when I got to the end of those it failed in another way.

Detected unique file "node_modules/argon2/bin/darwin-arm64-113" in "/Users/me/Documents/app/release/prod/mac-universal--arm64/app.app/Contents/Resources/app.asar" not covered by allowList rule: "undefined"

and then in Electron-Builder if I add that file to singleArchFiles then I get:

 Command failed: lipo /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/arm64-mw1MJx/node_modules/argon2/bin/darwin-arm64-113/argon2.node /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node -create -output /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node
fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/arm64-mw1MJx/node_modules/argon2/bin/darwin-arm64-113/argon2.node and /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node have the same architectures (arm64) and can't be in the same fat output file
  failedTask=build stackTrace=Error: Command failed: lipo /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/arm64-mw1MJx/node_modules/argon2/bin/darwin-arm64-113/argon2.node /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node -create -output /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node
fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/arm64-mw1MJx/node_modules/argon2/bin/darwin-arm64-113/argon2.node and /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node have the same architectures (arm64) and can't be in the same fat output file

[mmaietta]

This change seems the most likely, but I don't know what could be going wrong with those. @electron/universal was just a minor version jump

Can you provide a minimum repro repo/gist of the issue that I could check out locally? Can you try 24.1.2 as well? There was actually a native modules-related bug in 24.1.1

v23.6.0...v24.1.2

-    "@electron/universal": "1.2.1",
+    "@electron/notarize": "^1.2.3",
+    "@electron/osx-sign": "^1.0.4",
+    "@electron/rebuild": "^3.2.10",
+    "@electron/universal": "1.3.4",
-    "electron-osx-sign": "^0.6.0",

[Nantris]

Unfortunately 24.1.2 doesn't work either. I tried for a while to recreate this with electron-react-boilerplate but it uses a two-level package.json structure and it does manage to build properly.

Is there any template you can recommend for a repro? The only other template I generally use is electron-forge, but then it's not using electron-builder so that's no good.

I tried various versions of the dependency, I tried clearing our electron-builder config customizations until it was the minimum required. We have another native dependency (sharp) but it doesn't complain for that.

I guess I'll try pinning the @electron/universal dependency next.

Edit: Unfortunately reverting the versions of @electron/universal and electron-osx-sign has no effect - the same error occurs. 24.0.0-alpha.1 crashes just the same as the latest.

[Nantris]

Got a working (technically not working*) repro: https://github.com/Slapbox/repro-electron-builder-7512

[mmaietta]

Thanks for the link! I got the app launching with adding mergeASARs: false to mac config

[Nantris]

Ah jeez, yeah that does the trick but I had just needed to remove that to get things working with Electron-Builder 23 so I didn't think to add it back. Huge thank you!

Is there any reason that comes to mind why we would suddenly need to add this in 24 when we didn't need it before?

[mmaietta]

I'm not too sure tbh. The only culprit I can think of is that electron-builder was migrated to the official @electron/rebuild from the Go-based library that is used for building native modules. The migration was necessary in order to support the latest versions of electron though

[davej]

Just to chime in and say that we are noticing a very similar issue with 24.x.x. It seems to be related. When we switch back to 23.x.x then the issue goes away.

We are building non-universal apps. The app seems builds without error but the Mac x64 version of the app is damaged. There are no issues with the arm64 version.

When I verify the code signature then I get the following:

$ codesign --verify --deep --verbose=2 /Users/dave/Downloads/MyApp.app
<snip>
/Users/dave/Downloads/MyApp.app: a sealed resource is missing or invalid
file modified: /Users/dave/Downloads/MyApp.app/Contents/Resources/app.asar.unpacked/node_modules/node-mac-permissions/node-addon-api/nothing.target.mk

In our case, we can work around it for the moment with the '!node_modules/**/*.{mk,a,o,h}' hack.

[mmaietta]

@davej happy to take a look. I'm trying to figure out how to repro this. I'm wondering if !node_modules/**/*.{mk,a,o,h} should just be added by default, but I can't even repro the core issue locally.

Just tried locally on https://github.com/mmaietta/electron-builder-test with node-mac-permissions and tried building x64 on my m1 laptop. codesign didn't return any errors though.

Can you try creating a sample of the issue forking from here? https://github.com/mmaietta/electron-builder-test

[mmaietta]

@slapbox I'm diving in further and noticed something odd with 23.6.0 vs 24.x.x
The native modules assembled as part of app-builder-bin does not generate a .forge_meta file

 ~/D/repro-electron-builder-7512-master> rm -rf dist; yarn dist-mac        
yarn run v1.22.19
$ electron-builder --config ./electron-builder.js
  • electron-builder  version=23.6.0 os=22.4.0
....

~/D/repro-electron-builder-7512-master> ls **/node_modules/argon2/build/Release/.forge-meta
zsh: no matches found: **/node_modules/argon2/build/Release/.forge-meta

versus

yarn run v1.22.19
$ electron-builder --config ./electron-builder.js
  • electron-builder  version=24.1.2 os=22.4.0
....
⨯ Can't reconcile two non-macho files node_modules/argon2/build/Release/.forge-meta  failedTask=build stackTrace=Error: Can't reconcile two non-macho files node_modules/argon2/build/Release/.forge-meta
    at Object.exports.mergeASARs (/Development/repro-electron-builder-7512-master/node_modules/@electron/universal/src/asar-utils.ts:151:13)
    at exports.makeUniversalApp (/Development/repro-electron-builder-7512-master/node_modules/@electron/universal/src/index.ts:231:13)

....

~/D/repro-electron-builder-7512-master> ls **/node_modules/argon2/build/Release/.forge-meta
	dist/mac-universal-arm64-temp/secure-electron-template.app/Contents/Resources/app.asar.unpacked/node_modules/argon2/build/Release/.forge-meta
	dist/mac-universal-x64-temp/secure-electron-template.app/Contents/Resources/app.asar.unpacked/node_modules/argon2/build/Release/.forge-meta
	node_modules/argon2/build/Release/.forge-meta

The file is coming from here:
https://github.com/electron/rebuild/blob/29365ad904a8713567c4a7beb745d32df7f229e1/src/module-rebuilder.ts#L30

I need to figure out how disable these files being generated OR how electron-forge/packager is handling these files for universal builds

[sandeep1995]

Hi @mmaietta

a sealed resource is missing or invalid
file modified: /Users/dave/Downloads/MyApp.app/Contents/Resources/app.asar.unpacked/node_modules/node-mac-permissions/node-addon-api/nothing.target.mk

The issue can be reproduced if you have both the archs as target.

"mac": {
    "category": "public.app-category.utilities",
     "target": [
       {
         "target": "dir",
         "arch": [
           "x64",
           "arm64"
         ]
       }
   ],
  "icon": "build/icon.icns",
  "type": "distribution",
  "hardenedRuntime": true,
  "gatekeeperAssess": true
  }

[mmaietta]

Add this while I try and figure out what's going on here

     "!**/nothing.target.mk",
     "!**/.forge-meta",
     "!**/bin",
     "!node_modules/**/**.{mk,a,o,h,forge-meta}"

From what I can see, something is happening with the new electron/rebuild integration where all files are being packaged into each respective arch-specific .app before lipo/stitching together the .app
For node-mac-permissions, this is what I'm seeing

lipo -info dist/**/*.node                                                                 1 master!
Non-fat file: dist/mac-universal--arm64/electron-quick-start-typescript.app/Contents/Resources/app.asar.unpacked/node_modules/node-mac-permissions/bin/darwin-arm64-113/node-mac-permissions.node is architecture: arm64
Non-fat file: dist/mac-universal--arm64/electron-quick-start-typescript.app/Contents/Resources/app.asar.unpacked/node_modules/node-mac-permissions/bin/darwin-x64-113/node-mac-permissions.node is architecture: x86_64
Non-fat file: dist/mac-universal--arm64/electron-quick-start-typescript.app/Contents/Resources/app.asar.unpacked/node_modules/node-mac-permissions/build/Release/permissions.node is architecture: arm64
Non-fat file: dist/mac-universal--x64/electron-quick-start-typescript.app/Contents/Resources/app.asar.unpacked/node_modules/node-mac-permissions/bin/darwin-arm64-113/node-mac-permissions.node is architecture: arm64
Non-fat file: dist/mac-universal--x64/electron-quick-start-typescript.app/Contents/Resources/app.asar.unpacked/node_modules/node-mac-permissions/bin/darwin-x64-113/node-mac-permissions.node is architecture: x86_64
Non-fat file: dist/mac-universal--x64/electron-quick-start-typescript.app/Contents/Resources/app.asar.unpacked/node_modules/node-mac-permissions/build/Release/permissions.node is architecture: x86_64

For some reason both arm64 and x86_64 native modules are in mac-universal--x64 build, same with mac-universal--arm64. This is not the correct behavior

In 23.6.0 using app-builder-bin for native modules, there's no /node_modules/node-mac-permissions/bin/* folder

[mmaietta]

FOUND THE CULPRIT
https://github.com/electron/rebuild/blob/29365ad904a8713567c4a7beb745d32df7f229e1/src/module-rebuilder.ts#L111-L118

Seems there's two needed changes for electron-builder. FileMatcher needs these regexes by default

export const excludedExts = "iml,hprof,orig,pyc,pyo,rbc,swp,csproj,sln,suo,xproj,cc,d.ts," +
  // https://github.com/electron-userland/electron-builder/issues/7512
  "mk,a,o,h,forge-meta"

And electron/rebuild with disablePreGypCopy: true

Seems to work locally so far. Opened PR #7519