Faces: unsupported image type

[spicemint]

Describe the bug
When running the image classifier (doing it from cli) i get a bunch of errors

Classifier process output: Error: Expected image (BMP, JPEG, PNG, or GIF), but got unsupported image type
at getImageType (/var/lib/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/image.js:351:15)
at Object.decodeImage (/var/lib/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/image.js:196:21)
at main (/var/lib/nextcloud/apps/recognize/src/classifier_faces.js:64:28)

Result for IMG-20220119-WA0007.jpg = ["people"]

To Reproduce
Steps to reproduce the behavior:
run classifier (for images) from cli

Expected behavior
classify all images without any error, also would be great to know what image is creating the issue and why.

Recognize (please complete the following information):

• JS-only mode: no
• Enabled modes: object detection, face recognition

Server (please complete the following information):

• Nextcloud: 22.2.3
• OS: Arch Linux
• RAM: 64GB
• Processor Architecture x86

Additional context
latest recognize version 1.9.0

[marcelklehr]

The problematic image in this case is a contact photo of one of your users, which appears to be in a format that can't be handled. The app will ignore contacts with a photo in an unsupported format, though, and continue uninterrupted.

[spicemint]

i just ran it again, this time it completes without errors but says at the end:

No photos found of user David

David has a whole bunch of regular pictures and also contact pictures

[marcelklehr]

No photos found of user David
David has a whole bunch of regular pictures and also contact pictures

The app tags all photos it finds in the first run and will not touch them again in subsequent runs for performance reasons. If you want to reclassify files that have already been classified you'll need to reset the classifications in the app settings.

I've changed that message to be more clear :)

[spicemint]

thanks, is the same user that had that error with which I had started the threat.
When the user clicked in nextcloud on the folder that contains the pictures with tags it showed only tag names but empty folders, clicking a couple times on refresh finally changed something and content started showing up including pics, I need to check the logs and see if something shows up at that time, will report back

[spicemint]

earlier issue with empty folders does not happen anymore. I updated to nextcloud 23 and recognize 1.10.0 but new error for same user, error now:

classifying photos of user David using imagenet
Classifying array (
0 => '/var/nextcloud/David/files/SofortUpload/WhatsApp Images/IMG-20220128-WA0004.jpg',
1 => '/var/nextcloud/David/files/SofortUpload/WhatsApp Images/IMG-20220128-WA0006.jpg',
2 => '/var/nextcloud/David/files/SofortUpload/WhatsApp Images/IMG-20220128-WA0007.jpg',
)
Running array (
0 => '/var/lib/nextcloud/apps/recognize/bin//node',
1 => '/var/lib/nextcloud/apps/recognize/src/classifier_imagenet.js',
2 => '-',
)
Classifier process output: SyntaxError: Unexpected end of JSON input
at JSON.parse ()
at NodeFileSystem. (/var/lib/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/io/file_system.js:225:40)
at step (/var/lib/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/io/file_system.js:48:23)
at Object.next (/var/lib/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/io/file_system.js:29:53)
at fulfilled (/var/lib/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/io/file_system.js:20:58)

Classifier process output: SyntaxError: Unexpected end of JSON input
at JSON.parse ()
at NodeFileSystem. (/var/lib/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/io/file_system.js:225:40)
at step (/var/lib/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/io/file_system.js:48:23)
at Object.next (/var/lib/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/io/file_system.js:29:53)
at fulfilled (/var/lib/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/io/file_system.js:20:58)

Failed to classify images
Classifier process error

had a look at a pic, looks normal?

mediainfo "/var/nextcloud/David/files/SofortUpload/WhatsApp Images/IMG-20220128-WA0004.jpg"
General
Complete name : /var/nextcloud/David/files/SofortUpload/WhatsApp Images/IMG-20220128-WA0004.jpg
Format : JPEG
File size : 66.3 KiB

Image
Format : JPEG
Width : 847 pixels
Height : 1 024 pixels
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Compression mode : Lossy
Stream size : 66.3 KiB (100%)

[marcelklehr]

It seems like it's having trouble parsing the definition of the AI model. Can you try removing the models/ folder inside apps/recognize ? It will redownload the models from github then.

[spicemint]

done that, seemed to have helped but now back to original error:

Classifier process output: Error: Expected image (BMP, JPEG, PNG, or GIF), but got unsupported image type
at getImageType (/var/lib/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/image.js:351:15)
at Object.decodeImage (/var/lib/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/image.js:196:21)
at main (/var/lib/nextcloud/apps/recognize/src/classifier_faces.js:64:28)

Result for IMG-20220128-WA0004.jpg = []
Result for IMG-20220128-WA0006.jpg = []
Result for IMG-20220128-WA0007.jpg = []
Result for IMG-20220129-WA0000.jpg = []
Result for IMG-20220129-WA0001.jpg = ["people"]
Result for IMG-20220129-WA0002.jpg = ["people"]
Result for IMG-20220129-WA0003.jpg = []

but seems still to be normal pics, e.g.:

mediainfo "/var/nextcloud/David/files/SofortUpload/WhatsApp Images/IMG-20220128-WA0004.jpg"
General
Complete name : /var/nextcloud/David/files/SofortUpload/WhatsApp Images/IMG-20220128-WA0004.jpg
Format : JPEG
File size : 66.3 KiB

Image
Format : JPEG
Width : 847 pixels
Height : 1 024 pixels
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Compression mode : Lossy
Stream size : 66.3 KiB (100%)

[marcelklehr]

In #139 it transpired that this may be related to contact photos uploaded on iOS. Can you confirm that?

[spicemint]

it is odd, i just ran recognize again and now it finishes without a mistake and says user David has no unclassified pics, which i do not expect is the case, after re-run same result

as to your question, looked at the #139 my users have iOS only on their ipads, but no contacts were created on iOS, they got synced to iOS from gmail and nextcloud but the nextcloud app is also installed on the ipads so there could be some re-sync of contact with pictures that were in the gmail address book or on their android phones, is there a way i could check if it "came" from iOS?

about the face recognition, where are they put, when i open the "tags" in pictures on nextcloud i see all kind of tags but none that says e.g. faces

[marcelklehr]

says user David has no unclassified pics,

See above:

If you want to reclassify files that have already been classified you'll need to reset the classifications in the app settings.

On command line you can do

$ off recognize:reset-tags && occ recognize:classify-images

is there a way i could check if it "came" from iOS?

You can try overriding a contact picture using the nextcloud web UI. If that contact doesn't produce an error (in this case it could be verified as one error less perhaps) we know at least that the normal built-in functionality doesn't cause this.

about the face recognition, where are they put, when i open the "tags" in pictures on nextcloud i see all kind of tags but none that says e.g. faces

Pictures with any faces in them will get a "people" tag.