macOS Mojave and Catalina TCC trigger

[pnerum]

Description

When launching Processing.app and a user wants to use audio input for example, the app doesn’t aks for permission to use the microphone

Expected Behavior

Trigger a window asking the user for the processing.app to access the microphone

Current Behavior

No tcc triggers, so there are no audio or video inputs available

Steps to Reproduce

1. Open Processing.app
2. Import sound library
3. Open examples -->Libraries-->Sound-->IO-->AudioInput

Your Environment

• Processing version: 3.5.4 and 4.0 beta
• Operating System and OS version: 10.14.6

Possible Causes / Solutions

Use the Apple provided API call to add some tags in the Info.plist file.
https://developer.apple.com/documentation/avfoundation/cameras_and_media_capture/requesting_authorization_for_media_capture_on_macos

[pnerum]

I have a Developer ID myself and could perform some more tests.
In the Processing.app bundle you added the "jdk1.8.0_202.jdk”. This bundle has to have the same changes as noted in my first request. I edited the "Processing.app/Contents/PlugIns/jdk1.8.0_202.jdk/Contents/Info.plist” to add these two keys:
<key>NSCameraUsageDescription</key>
<string>Processing Java needs access to the camera for your video calls</string>
<key>NSMicrophoneUsageDescription</key>
<string>Processing Java needs access to the microphone for your calls</string>
Resigned that Bundle with my own Developer ID and it worked!

[pnerum]

[kevinstadler]

@pnerum does the workaround described in #30 work for you?

In order to fix this in the library itself (without requiring the entire Processing app to ask for permissions on startup which it might not end up using) we'd need to ask for the permission programmatically near the code block below, where there's currently only an if statement handling audio permissions on Android. @pnerum since I can't test this myself, can you tell me if:

1. it is possible to request the audio-in permission programmatically only when it's actually needed during runtime?
2. how can we detect reliably whether a TCC trigger is necessary? In the code below it's the class type of the audio manager that tells us we need to test for permission, but maybe in this case a call to read out some system/OS constants will be necessary, do you have a hunch which?
   

   processing-sound/src/processing/sound/AudioIn.java

   Lines 44 to 58 in 7a56aa4

   	if (Engine.getAudioManager() instanceof JSynAndroidAudioDeviceManager) {
   	if (in != 0) {
   	Engine.printWarning("if you want to capture audio from somewhere other than the default\n" +
   	"device on Android, use: new Sound(this).inputDevice(deviceID)\n" +
   	"where for deviceID you can fill in any of Android's MediaRecorder.AudioSource constants.");
   	}
   	// we're on Android, check if the sketch has permission to capture Audio
   	if (!parent.hasPermission(Manifest.permission.RECORD_AUDIO)) {
   	Engine.printError(AudioIn.ANDROID_PERMISSION_WARNING_MESSAGE);
   	throw new AndroidPermissionException("RECORD_AUDIO permission not granted");
   	}
   	Engine.printMessage("capturing audio in from device " + Engine.getSelectedInputDeviceName());
   	}

[pnerum]

Hi Kevin,
In my testing it was only asking for permission when I started the AudioInput application (in runtime) and not by starting Processing.app That’s why Processing.app/Contents/PlugIns/jdk1.8.0_202.jdk/Contents/Info.plist needs the plist to be edited and signed again. It’s the java binary in Processing.app/Contents/PlugIns/jdk1.8.0_202.jdk/Contents/Home/jre/bin/java that’s asking for permission to use the microphone.

As you can see in the logstream output:

2020-01-27 11:13:02.659628+0100 0x1280245 Info 0xe96bc 281 0 tccd: [com.apple.TCC:access] AttributionChain: RESP:{ID: ??, PID[75127], auid: 501, euid: 501, responsible path: '/Users/testing/Desktop/Processing.app/Contents/MacOS/Processing', binary path: '/Users/testing/Desktop/Processing.app/Contents/MacOS/Processing'}, ACC:{ID: net.java.openjdk.cmd, PID[75139], auid: 501, euid: 501, binary path: '/Users/testing/Desktop/Processing.app/Contents/PlugIns/jdk1.8.0_202.jdk/Contents/Home/jre/bin/java'}, REQ:{ID: com.apple.audio.coreaudiod, PID[170], auid: 202, euid: 202, binary path: '/usr/sbin/coreaudiod'}

[kevinstadler]

Oh great! Could you do me a favour and install Minim (an alternative sound library) from within Processing and see if it behaves in the same way when you run its audio-in example?

If the behaviour is the same and the extra permission entry doesn't interfere with Processing startup until a sketch actually attempts to access the microphone then @benfry I think this should be fixed at the level of the Processing app itself?

I don't know how exactly the whole .App bundling thing works but maybe the permissions need to be added here? https://github.com/processing/processing/blob/349f413a3fb63a75e0b096097a5b0ba7f5565198/java/application/Info.plist.tmpl

[pnerum]

Just tested it and Minim does indeed behaves exactly the same as the audioin example.

[pnerum]

Pay attention as the text string of my example should say something more logical. I copied those strings from Skype.app, just for testing purposes.

<string>Processing Java needs access to the camera for your video calls</string>
<string>Processing Java needs access to the microphone for your calls</string>

[kevinstadler]

@pnerum thanks so much, that's a huge help!

@benfry could this be moved back to (and addressed in) the main processing repository?

[pnerum]

I’ve added the examples: https://gist.github.com/pnerum/75a7e56edf545cb3114d5794a53b8699.js

[pnerum]

@kevinstadler No problem! If this gets solved our students and staff will be very happy!

[pnerum]

@kevinstadler and @benfry isn’t this issue in the wrong project?
The TCC trigger also needs to be triggered for video. I just performed a test as you can see in my screenshot.

[pnerum]

Hi @kevinstadler the issue isn’t solved until someone from the developers need to edit the Info.plist and redistribute this updated app version as a new release.
See: #51 (comment)