M1 Issues

[tomekkosinski]

MacOS version: 12.4 /
Chip: Apple M1 Pro
Python ARM compilation

DelphiFMX runs smoothly on Python 3.8. Hower it hungs on Python 3.10 just upon importing:
"from delphifmx import*"

[lmbelo]

Hi @cosina,

This is a known bug in M1. We are still working in a solution.

[tomekkosinski]

I just did more test and the last version it works is Python 3.8.9

Problem does not seems to be related to M1 generation only because running 3.9 Inte (but on M1 machine) falied as well.

[tomekkosinski]

I dig a bit more: MacOS Monteray has a build in 3.8.9. ANd this version is DIFFERENT than 3.8.9 downloaded from python.org. The main difference seems to be that the built-in version is a framework-based version which enables GUI applications. Therefore as long as I use a built-in python3 everything works, but using any version from python.org fails.

See: https://github.com/python/cpython/blob/main/Mac/README.rst

[lmbelo]

Hello @cosina,

Can you share more details? It would be good if you share screenshots, step-by-step to reproduce and a sample script.
If I didn't get it wrong, this is related to the dynamic linker loader deadlock that crops up from time-to-time.
It might work eventually. I realized that it is working well on Conda.

Run "kill -ABRT {PID}" to send a SIGABRT when your app hangs to generate a crash report.

Thread 0 is stuck waiting for thread 1 to finish but thread 1 can’t finish because thread 0 is holding a lock that it needs to acquire. See it bellow:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x189a7370c __ulock_wait + 8
1 libdispatch.dylib 0x1898ef5a0 _dlock_wait + 56
2 libdispatch.dylib 0x1898ef4d0 _dispatch_once_wait + 120
3 CoreFoundation 0x189b4ab2c CFURLCopyResourcePropertyForKey + 284
4 HIToolbox 0x192945350 ___HIMagnifiedMode_block_invoke + 104
5 libdispatch.dylib 0x1898eebac _dispatch_client_callout + 20
6 libdispatch.dylib 0x1898f0454 _dispatch_once_callout + 32
7 HIToolbox 0x1926e88bc _HIMagnifiedMode + 64
8 AppKit 0x18c69bc08 -[NSScreen backingScaleFactor] + 40
9 AppKit 0x18c69b450 _NSScreenConfigurationUpdateWithSharedInfo + 2072
10 AppKit 0x18c699428 ___NSScreenConfigurationEnsureInitialUpdateOccurred_block_invoke + 124
11 libdispatch.dylib 0x1898eebac _dispatch_client_callout + 20
12 libdispatch.dylib 0x1898f0454 _dispatch_once_callout + 32
13 AppKit 0x18c6e042c +[_NSScreenConfiguration latestScreens] + 280
14 AppKit 0x18c6e01d8 +[NSScreen mainScreen] + 32
15 libmyguilib.dylib 0x109633414 DispatchToImport + 160
16 libmyguilib.dylib 0x109989a88 System::Mac::Devices::AddDevices() + 84
17 libmyguilib.dylib 0x109987adc System::Devices::TDeviceInfo::operator cctr() + 256
18 libmyguilib.dylib 0x10968a588 System::InitUnits() + 252
19 libmyguilib.dylib 0x10968a7e0 System::_StartLib(System::TInitContext*, System::TLibModule*, void ()(int, void)) + 412
20 libmyguilib.dylib 0x109674d64 Sysinit::_InitLib(System::TInitContext*, void*) + 412
21 libmyguilib.dylib 0x10a0494a8 Myguilib::initialization() + 64
22 dyld 0x105619c1c invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 164
23 dyld 0x105642d98 invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 168
24 dyld 0x105639988 invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 532
25 dyld 0x105605f98 dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 168
26 dyld 0x10563972c dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 192
27 dyld 0x1056426c0 dyld3::MachOAnalyzer::forEachInitializerPointerSection(Diagnostics&, void (unsigned int, unsigned int, unsigned char const*, bool&) block_pointer) const + 148
28 dyld 0x1056429c0 dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 432
29 dyld 0x105619b5c dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 172
30 dyld 0x105619d00 dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 208
31 dyld 0x105619dcc dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 124
32 dyld 0x105629734 dyld4::APIs::dlopen_from(char const*, int, void*) + 512
33 Project1 0x104f5927c Project1::main() + 28 (Project1.dpr:12)
34 Project1 0x104f59074 main + 96 (Project1.dpr:22)
35 dyld 0x1056090f4 start + 520

Thread 1:: Dispatch queue: com.apple.root.utility-qos.overcommit
0 libsystem_kernel.dylib 0x189a7370c __ulock_wait + 8
1 libsystem_platform.dylib 0x189ac2140 _os_unfair_lock_lock_slow + 228
2 dyld 0x105629620 dyld4::APIs::dlopen_from(char const*, int, void*) + 236
3 CoreFoundation 0x189b4add4 __CFLookupCoreServicesInternalFunction + 76
4 CoreFoundation 0x189b4ad78 ____CFCoreServicesInternal__FSURLCopyResourcePropertyForKey_block_invoke + 24
5 libdispatch.dylib 0x1898eebac _dispatch_client_callout + 20
6 libdispatch.dylib 0x1898f0454 _dispatch_once_callout + 32
7 CoreFoundation 0x189b4ab2c CFURLCopyResourcePropertyForKey + 284
8 CoreFoundation 0x189b4a2b0 ____CFRunLoopSetOptionsReason_block_invoke_5 + 180
9 libdispatch.dylib 0x1898ece60 _dispatch_call_block_and_release + 32
10 libdispatch.dylib 0x1898eebac _dispatch_client_callout + 20
11 libdispatch.dylib 0x189900554 _dispatch_root_queue_drain + 964
12 libdispatch.dylib 0x189900b58 _dispatch_worker_thread2 + 164
13 libsystem_pthread.dylib 0x189aa92c8 _pthread_wqthread + 228
14 libsystem_pthread.dylib 0x189aa8018 start_wqthread + 8

Thread 2:
0 libsystem_pthread.dylib 0x189aa8010 start_wqthread + 0

Thread 3:
0 libsystem_kernel.dylib 0x189a71954 mach_msg_trap + 8
1 libsystem_kernel.dylib 0x189a71d00 mach_msg + 76
2 Project1 0x104dfaba4 System::Internal::Machexceptions::ExcThread(void*) + 160 (System.Internal.MachExceptions.pas:240)
3 libsystem_pthread.dylib 0x189aad240 _pthread_start + 148
4 libsystem_pthread.dylib 0x189aa8024 thread_start + 8

Check out this thread for similar issues: https://developer.apple.com/forums/thread/695702?answerId=696925022#696925022

[tomekkosinski]

Hi,
I'm more than sure that it is related only to the version of Python. Embeded Python 3.8.9 was compiled with GUI support.
So as lon as I use /user/bin/python3 (wich should be 3.8.9) everything works as expected.
Your log is about getting access to a Screen object wich causes a deadlock due to missed GUI attribute of python3 itself.

See the image and used Python version details

I tested it also on my older, Intel based Mac - same conclusion.

[lmbelo]

Great, let's try it :)

[lmbelo]

Maybe we are talking about different issues. Give us more details, please.

[tomekkosinski]

OK. Let's systemize this

1. I tested DelphiFMX on 3 Macs. One Intel-based and two M1-based,
2. On Intel, everything works, but on M1 Python hung just on the "import delphifmx" statement.
3. So I jumped to conclusions that the problem is M1 related, and I filled the report,
4. I got info that it is a known M1 problem.
5. I dug deeper to find the problem is related to a Python version. On v3.8.9 it works but not on a newer version,
6. Then I found that it is not about the version but that the only Python3, which is a part of Monteray distribution, is compiled the way that can run GUI application. To be clear GUI means only Aqua access - XServers works as expected
7. See: https://github.com/python/cpython/blob/main/Mac/README.rst
8. I also did some debugging on loading DelphiFMXdylib from the command line application to discover that it hung upon accessing the Screen object. It is consistent with (7) explanation.
9. Final conclusion: to run DelphiFMX on Mac, use build-in /usr/bin/python3 version only.

[lmbelo]

I understand it works when GUI is loaded by Python. Thus, this is a workaround for the real issue.
If you run a GUI Delphi app and load a GUI shared library, it will work as well. If you load it on a console application it won't. GUI will be loaded by the dynamic linker loader and it will be trapped by a deadlock.
That's why I'm considering this a M1 issue rather then Python.

[tomekkosinski]

Hi,
You are absolutely right! After some updated, Python on macOS does not work anymore for me.

However, it seems a found a workaround. The problem is indeed a deadlock on TNSScreen.Wrap(TNSScreen.OCClass.mainScreen); mainly because it is done on library initialization.

I create a utility package:

// Init screen context on MacOS
library macosfmxrunner;

uses
  System.SysUtils,
  System.Classes,
  Macapi.AppKit,
  PythonEngine;

var
  gEngine : TPythonEngine;
  gModule : TPythonModule;

function PyInit_tatukgis_macos: PPyObject;
begin
  try
    Result := nil ;

    gEngine := TPythonEngine.Create(nil);
    gEngine.AutoFinalize := False;
    gEngine.UseLastKnownVersion := True;
    // Adapt to the desired python version - Will only work with this version

    gModule := TPythonModule.Create(nil);
    gModule.Engine := gEngine;
    // This must match the ProjectName and the function name pattern
    gModule.ModuleName := 'macosfmxrunner';

    gEngine.LoadDllInExtensionModule();
    Result := gModule.Module;

    // this line did the trick !
    TNSScreen.Wrap(TNSScreen.OCClass.mainScreen);
  except
    on E: Exception do begin
      WriteLn('An error has occurred: ' + E.Message);
    end;
  end;
  end;

exports
  PyInit_tatukgis_macos;

begin
end.

If I call:

import macosfmxrunner
import delphidmx

everything seems to work nicely. Furthermore, problems with text entering also seem to be resolved.

Tomek

[lmbelo]

Hi @cosina!

Thank you for your contribution.

• I'm considering that you're not running a Python compilation with GUI initialized.

The dynamic linker first runs the initializers. After that, Python will consume the exported method (the PPyObject as module). So, in theory, that wouldn't solve the deadlock issue, because it doesn't affect the code execution order. Maybe it may work at times and fail at others.

I bet the first initializer running in your code is the TScreen one, that's why you first see the TNSScreen. But, there are others to be loaded as well.

Maybe the initializers were sorted, maybe it was purely luck (hahaa), it's a deadlock...

[tomekkosinski]

Maybe pure luck. I use the official delphifmx build from PyPi.

Calling this code after library initialization force the app to initialize GUI context before importing delphifmx

At least I have more luck than ever running Python distributions from python.oirg

I will test it on several different Macs in the coming days. I also set test runners... so let's see.

[lmbelo]

Calling this code after library initialization force the app to initialize GUI context before importing delphifmx

Well, the first step in library loading (after loading symbols to memory) is performed by the dynamic linker, which is the "initialization". At this moment, every code placed into the Delphi "initialialization" section will run. So, the "PyInit_tatukgis_macos" will only be triggered after we have the library loaded and initialized.

Did you get the point?