Not detecting expected APIs

[gavrichards]

Hey - thank you for your work on this tool.

I've been receiving emails from Apple after submitting my app recently, regarding "Missing API declaration", so I'm hopeful this tool will help me to resolve it.

I've installed it and run it, but it's not giving me the output I would expect. Apple's email lists 4 missing API declarations, but when I run this script I'm told there were no reasons provided. I suspect it's not scanning my Pods files perhaps?

me@mbp ios % privacy-manifest analyze --project ./myapp.xcodeproj --reveal-occurrences --output ./

Xcode project detected: /Users/gav/Repos/myapp/ios/myapp.xcodeproj
---
✔ Parsed myapp's source files
✔ Parsed myapp-tvOS's source files
✔ Parsed myapp's Frameworks Build Phase
---
Execution took 0.030808667 seconds
---
User defaults APIs (0 occurrences)
Contacts (0 occurrences)
Third-party SDKs (0 occurrences)
Core Location (0 occurrences)
HealthKit (0 occurrences)
Active keyboard APIs (0 occurrences)
Disk space APIs (0 occurrences)
System boot time APIs (0 occurrences)
File Timestamp APIs (0 occurrences)
Crash data (1 occurrence)

	✎ /Users/gav/Repos/myapp/ios/AppDelegate.swift:
		9:	import Firebase

---

No reasons were provided, Privacy Manifest file generation was skipped.

Do you have any advice? Thank you.

[stelabouras]

Yes, try to either point the --project argument to the .xcworkspace container file (as this is the one that also includes the Pods Xcode project) or just to the current directory (--project .) where the tool will attempt to detect and prioritize the xcworkspace over the xcodeproj one.

Let me know if the tool detects those missing SDKs. If not, you can post those 4 missing API declarations and I will check if I need to add them to the internal list or make the tool look deeper into the source code.

[gavrichards]

I did originally do --project . but it ended up running for over an hour before I decided to terminate it as I had no idea how long it was going to take.

The declarations are:

• NSPrivacyAccessedAPICategoryUserDefaults
• NSPrivacyAccessedAPICategorySystemBootTime
• NSPrivacyAccessedAPICategoryDiskSpace
• NSPrivacyAccessedAPICategoryFileTimestamp

Thank you

[stelabouras]

Oh wow, taking an hour to process the workspace is indeed a really long time.

The tool works concurrently to speed up the process to a few seconds, but over an hour probably means that something is going wrong with the parsing or that there are a lot of dependencies to go through in the Pods project.

The declarations that you posted are exactly the ones configured to be detected by the tool. You don't have any references to NSUserDefaults / UserDefaults inside the source files of myapp.xcodeproj project?

I will have to investigate why the process takes so much time to complete. Unless the CocoaPods dependencies result into too many source files for processing, then there's probably a thread-locking issue lurking somewhere I haven't found yet.

[gavrichards]

You don't have any references to NSUserDefaults / UserDefaults inside the source files of myapp.xcodeproj project?

I just used my code IDE to search this directory, and no references to "UserDefaults" could be found.

[stelabouras]

I just used my code IDE to search this directory, and no references to "UserDefaults" could be found.

Interesting; then those "Missing API declarations" that Apple reports were found in the Pods project.

Do you maintain parts of your application's logic in separate Pods, or do you use CocoaPods only for third part libs? If it's the latter, and you don't have an issue sharing your Podfile, you can post it here so that I can try and recreate the issue when I can?

[stelabouras]

I think I have found the issue! I will issue a fix in a while and will ping you to test it!

[gavrichards]

Oh, great, thank you. I can certainly provide my Podfile if you need it, although I'd prefer not to post it publicly.

[Maker-Mark]

@stelabouras I'm having the same issue when trying to use the tool for the entire workspace, it infinitely runs.
@gavrichards In the interim, this tool was relatively helpful https://github.com/Wooder/ios_17_required_reason_api_scanner

[Maker-Mark]

@stelabouras Also, as noted by apple here https://developer.apple.com/documentation/bundleresources/privacy_manifest_files/describing_data_use_in_privacy_manifests

Third-party SDKs need to provide their own privacy manifest files that record the types of data they collect. Your app’s privacy manifest file doesn’t need to cover data collected by third-party SDKs that your app links to.

Would this tool be generating the declarations for 3rd party libraries? While adding the reasons to your app's manifest fixes the warning as of now, that does not seem complaint/may be an issue if apple starts checking who's declaring the reasons.

An example:
When I run this tool for my project file, I get 0 required API usages. That makes sense since my app's code does not use any of the APIs, but my code uses libraries that do. So shouldn't I just need to update the packages that are missing the declaration, and not make my own?

[gavrichards]

The problem is from Apple's email there's no way to actually know which third party library is using the permissions that are listed which needs declaring. I've got no idea which library(s) are using the 4 listed in the emails I've got from Apple.

As it's then up to that third party to update their code to declare the reasons for the permissions they're using, and they may not do that before the deadline, we could be in a situation where we have to declare it in our own app's privacy file for the time being otherwise we can't update our app.

[Maker-Mark]

@gavrichards Right, but I suspect apple will soon start to detect when you're declaring for 3rd party libraries. See the bolded part of the text.

Looking at the docs for the RequiredReasonAPI closer, I see

For each category of required reason API that your app or third-party SDK uses, add a dictionary to the NSPrivacyAccessedAPITypes array in your app or third-party SDK’s privacy manifest file that reports the reasons your app uses the API category. If you use the API in your app’s code, then you need to report the API in your app’s privacy manifest file. If you use the API in your third-party SDK’s code, then you need to report the API in your third-party SDK’s privacy manifest file. Your third-party SDK can’t rely on the privacy manifest files for apps that link the third-party SDK, or those of other third-party SDKs the app links, to report your third-party SDK’s use of required reasons API.

If I'm understanding it correctly, it's saying 3rd party SDK's cannot rely on the manifest from the app using it. If that's the case, as soon as they can detect that we're doing that, we'll see these issues again.

Reading this, it seems the actual way to become "complaint" is to add/update the manifest to each 3rd party SDK. But obviously, this is not sustainable, nor really something we can really know the reasons without being a maintainer of the SDK

[stelabouras]

Oh, great, thank you. I can certainly provide my Podfile if you need it, although I'd prefer not to post it publicly.

No need! I have just pushed 0.0.17 that hopefully fixes the stalling issue. Let me know how that goes. Same for @Maker-Mark.

In the interim, this tool was relatively helpful https://github.com/Wooder/ios_17_required_reason_api_scanner

Nice one! I had no idea that such tool existed for so long! I will give it a try as well.

Would this tool be generating the declarations for 3rd party libraries? While adding the reasons to your app's manifest fixes the warning as of now, that does not seem complaint/may be an issue if apple starts checking who's declaring the reasons.

An example:
When I run this tool for my project file, I get 0 required API usages. That makes sense since my app's code does not use any of the APIs, but my code uses libraries that do. So shouldn't I just need to update the packages that are missing the declaration, and not make my own?

I think that falls under the "third-party SDK requirements" document that Apple posted here where only for the listed SDKs (that the tool checks against) you are required to update them to a version where a Privacy Manifest file exists. But to answer your initial question, my tool does not generate declarations for 3rd party libraries, but it does provide a warning so that the developer can use to check the affected libraries more closely, like the following:

WARNING: The following third-party SDKs were detected:

* Found Reachability.
* Found SwiftProtobuf.
* Found lottie-ios.

ℹ---------------------------------------------------------------------------------+
| You must include the privacy manifest for any of the above SDKs when you submit |
| new apps in App Store Connect that include those SDKs, or when you submit an    |
| app update that adds one of the listed SDKs as part of the update.              |
| Signatures are also required in these cases where the listed SDKs are used      |
| as binary dependencies. Any version of a listed SDK, as well as any SDKs that   |
| repackage those on the list, are included in the requirement.                   |
+---------------------------------------------------------------------------------+

⚓︎ https://developer.apple.com/support/third-party-SDK-requirements/

[Maker-Mark]

@stelabouras Ahh okay thanks. Trying the update now.
Edit: I see an error
✖ Parser Error: The project cannot be found at /Users/me/src/github.com/org/mobile/ios/Pods/Pods.xcodeproj

[gavrichards]

It seems to be working for me now, running:

% privacy-manifest analyze --project . --reveal-occurrences --output . --deep-library-framework-check

I'm just at a point where I need to make a selection. I don't understand this yet but there's some reading to do.
We do appear to be through the endless loop issue though, thank you!

[stelabouras]

@Maker-Mark Does this path exist? Is the Pods.xcodeproj project container on a different path? Was this error not occurring before 0.0.17?

[stelabouras]

@gavrichards I have tried to provide relevant links in the output of the CLI so that you can open them in a browser and check the Apple documentation on those APIs.

Note that for the APIs covered here such as location data, personal data, contacts, crashes etc, even though the tool attempts to detect as many as it can, the logic does not set any values on the generated Privacy Manifest file.

[Maker-Mark]

@stelabouras No, that path does not exist! I got that issue when running
privacy-manifest analyze --project ./ios/my_workspace.xcworkspace
Output:

Xcode workspace detected: /Users/me/src/github.com/privacy-manifest/ios/my_workspace.xcworkspace
✖ Parser Error: The project cannot be found at /Users/me/src/github.com/org/mobile/ios/Pods/Pods.xcodeproj

However, running the command @gavrichards did is working for me now and showing what I expect, thanks 🙏

[stelabouras]

@Maker-Mark That's strange. When you run the command with --project ., does the full path that gets printed at the first line of the output match the path that gets printed when passing --project ./ios/my_workspace.xcworkspace (namely /Users/me/src/github.com/privacy-manifest/ios/my_workspace.xcworkspace)?

[Maker-Mark]

@stelabouras Yes, it shows the repo's full directory
ie Directory detected: /Users/me/src/github.com/org/repo

[stelabouras]

Ah ok, so what happens is that when you pass --project . the tool tries to find either a .xcodeproj or a .xcworkspace at the specified directory. If not, then it treats it as a directory and parses all the supported files recursively.

The parser error is still baffling though: I would assume that this .xcworkspace that you specified before contains a reference to this Pods.xcodeproj that does not exist either because you haven't run the pod install or due to another issue (e.g. an out of date reference).

[stelabouras]

Closing the issue as both of you have reported that the tool now works.