Contact Pictures 401 Unauthorized

[james-choncholas]

Describe the bug
Face recognition fails to pull contact pictures with a 401 unauthorized response, shown in nextcloud logs. The failing URL in the log file correctly returns the contact image when visited from the browser.

At fist I thought this might be the fault of my reverse proxy (Traefik) so I changed the nextcloud configuration to 'overwrite.cli.url' => 'https://sub.domain.com/' to http://localhost/ but still see the 401 response.

Interestingly, when file_get_contents() fails I see the exact same behavior as issue #137 when running php occ recognize:classify-images from the cli. The 401 errors only appear in the nextcloud log, but the command output looks the same.

Stack trace from log:

{
  "reqId": "geL7jQxgMIf9MgDiVELz",
  "level": 3,
  "time": "2022-01-10T06:29:01+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "PHP",
  "method": "",
  "url": "--",
  "message": "file_get_contents(http://sub.domain.com/remote.php/dav/addressbooks/users/jim/personal/B8E3244B-F48C-43AB-A80C-FCDA482FF497.vcf?photo): Failed to open stream: HTTP request failed! HTTP/1.1 401 Unauthorized\r\n at /var/www/html/custom_apps/recognize/lib/Service/ReferenceFacesFinderService.php#72",
  "userAgent": "--",
  "version": "22.2.3.0",
  "exception": {
    "Exception": "Error",
    "Message": "file_get_contents(http://sub.domain.com/remote.php/dav/addressbooks/users/jim/personal/B8E3244B-F48C-43AB-A80C-FCDA482FF497.vcf?photo): Failed to open stream: HTTP request failed! HTTP/1.1 401 Unauthorized\r\n at /var/www/html/custom_apps/recognize/lib/Service/ReferenceFacesFinderService.php#72",
    "Code": 0,
    "Trace": [
      {
        "function": "onError",
        "class": "OC\\Log\\ErrorHandler",
        "type": "::",
        "args": [
          2,
          "file_get_contents(http://sub.domain.com/remote.php/dav/addressbooks/users/jim/personal/B8E3244B-F48C-43AB-A80C-FCDA482FF497.vcf?photo): Failed to open stream: HTTP request failed! HTTP/1.1 401 Unauthorized\r\n",
          "/var/www/html/custom_apps/recognize/lib/Service/ReferenceFacesFinderService.php",
          72
        ]
      },
      {
        "file": "/var/www/html/custom_apps/recognize/lib/Service/ReferenceFacesFinderService.php",
        "line": 72,
        "function": "file_get_contents",
        "args": [
          "http://sub.domain.com/remote.php/dav/addressbooks/users/jim/personal/B8E3244B-F48C-43AB-A80C-FCDA482FF497.vcf?photo"
        ]
      },
      {
        "file": "/var/www/html/custom_apps/recognize/lib/Service/ClassifyImagesService.php",
        "line": 85,
        "function": "findReferenceFacesForUser",
        "class": "OCA\\Recognize\\Service\\ReferenceFacesFinderService",
        "type": "->",
        "args": [
          "jim"
        ]
      },
      {
        "file": "/var/www/html/custom_apps/recognize/lib/BackgroundJobs/ClassifyImagesJob.php",
        "line": 64,
        "function": "run",
        "class": "OCA\\Recognize\\Service\\ClassifyImagesService",
        "type": "->",
        "args": [
          "jim",
          100
        ]
      },
      {
        "file": "/var/www/html/lib/public/BackgroundJob/Job.php",
        "line": 79,
        "function": "run",
        "class": "OCA\\Recognize\\BackgroundJobs\\ClassifyImagesJob",
        "type": "->",
        "args": [
          null
        ]
      },
      {
        "file": "/var/www/html/lib/public/BackgroundJob/TimedJob.php",
        "line": 63,
        "function": "execute",
        "class": "OCP\\BackgroundJob\\Job",
        "type": "->",
        "args": [
          {
            "__class__": "OC\\BackgroundJob\\JobList"
          },
          {
            "__class__": "OC\\Log"
          }
        ]
      },
      {
        "file": "/var/www/html/cron.php",
        "line": 127,
        "function": "execute",
        "class": "OCP\\BackgroundJob\\TimedJob",
        "type": "->",
        "args": [
          {
            "__class__": "OC\\BackgroundJob\\JobList"
          },
          {
            "__class__": "OC\\Log"
          }
        ]
      }
    ],
    "File": "/var/www/html/lib/private/Log/ErrorHandler.php",
    "Line": 92,
    "CustomMessage": "--"
  },
  "id": "61dc621541d41"
}

To Reproduce
Steps to reproduce the behavior:

1. Install recognize from Nextcloud Apps
2. Enable face recognition
3. Wait for cron or run cli (both fail)

Expected behavior
Contact faces correctly recognized without 401 errors in log.

Recognize (please complete the following information):

• JS-only mode: No
• Enabled modes: object detection, face recognition

Server (please complete the following information):

• Nextcloud: 22.2.3.0
• OS: Official Nextcloud docker container (Ubuntu host)
• RAM: 32GB
• Processor Architecture: x86

Additional context
Configuration 'allow_local_remote_servers' => true, doesn't appear to make a difference.
Object recognition works as expected.

[marcelklehr]

Interesting!

It appears from this comment in nextcloud/mail that prior to contacts app v1.0 photos would be stored/served as URLs, while afterwards they are stored/served as binary blobs. I'm not sure if this is the whole story, though. It could also be that photos are cached somewhere and henceforth returned as URLs to save bandwidth.

[marcelklehr]

Can you fetch http://sub.domain.com/remote.php/dav/addressbooks/users/jim/personal/B8E3244B-F48C-43AB-A80C-FCDA482FF497.vcf (without the photo suffix) to see what the contact card looks like (especially the format of the photo).

[james-choncholas]

Fetching the vcf directly, it appears to have the jpg encoded directly into the text. The line looks like:

PHOTO;ENCODING=b;TYPE=JPEG;X-ABCROP-RECTANGLE=ABClipRect_1&0&45&267&267&DeEk2UvjfOKNXsgfSDpI0g==:
...jpeg blob...

I created the contact a long time ago, it may have been with the older version of the contacts app, but I added the contact image using iOS just a few days ago.

It appears the mail app just skips those photos where I'd like recognize to use them. Do you think reinstalling the latest contacts app, cleaning up the database clutter it left behind, then re-uploading contacts would be the easiest fix?

[james-choncholas]

Before going through the trouble of re-installing the contacts app, I ran a test with a brand new contact and photo (created from iOS.) Recognize failed in the same way with the 401 for the new contact.

[marcelklehr]

@ChristophWurst Do you have some insight here maybe or know who might?

[james-choncholas]

It seems this issue only happens with contacts created from iOS. I tested by creating a new contact using the nextcloud contacts app and recognize does not log any errors for it.

What's more, the test contact image (which did not fail with 401) was the exact same as the photo I ran the classifier on, but nothing was recognized. Could the other failing contacts from iOS be preventing the classifier_faces from completing? The code seems to gracefully log the error and continue but maybe I missed something.

The error logged for the failing iOS contacts looks like:

Classifier process output: Error: Expected image (BMP, JPEG, PNG, or GIF), but got unsupported image type                                                                                                       
    at getImageType (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-node/dist/image.js:351:15)                                                                                               
    at Object.decodeImage (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-node/dist/image.js:196:21)                                                                                         
    at main (/var/www/html/custom_apps/recognize/src/classifier_faces.js:64:28)

[marcelklehr]

It seems this issue only happens with contacts created from iOS

interesting!

The failing contacts will cause the classification run to fail in the end, but everything that can be classified should still be classified. Do you get a 'people' tag on images with a face? (You should)

the exact same as the photo I ran the classifier on

Did you run the classifier using occ or directly using node.js?

[james-choncholas]

Do you get a 'people' tag on images with a face?

Yes! It seems I was wrong, ConactManager returns URIs for photos and classify_faces gracefully logs the error and continues.

Did you run the classifier using occ or directly using node.js?

occ

Here's a question. How does authentication happen when recognize fetches a contact photo from a URI? Authentication is required otherwise anyone could access contact photos via the URI, right? Should recognize be attaching a Authorization header when calling get_file_contents?

[marcelklehr]

ConactManager returns URIs for photos and classify_faces gracefully logs the error and continues

ok, that's good

Should recognize be attaching a Authorization header when calling get_file_contents?

That would be the fix, yeah. I'm just wondering if this URL thing is at all intentional on the part of Nextcloud/Contacts.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.