IFilesMetadata

[ArtificialOwl]

#40676

Introduction

This is a feature that allow any app to store files metadata.

When a file is created or modified, an event is broadcasted so that a listener can:

• request for an identical event, but on the background job process
• set metadata, key and value
• define metadata index
• help building queries to search on an indexed metadata
• help building queries that returns metadata linked to fileid

tasks

• model Metadata
  • store type of metadata (string, int, bool, array, float)
  • serialize/deserialize and import from database
• store metadata,
  • update database only if new data (or truly edited) is available
  • avoid parallel update in database
  • delete metadata based on key
  • delete metadata based on prefix
• index metadata (file id, metadata key, metadata value)
  • generate entries in metadata_index based on value type
• queries
  • helper to left join metadata based on file_id
  • helper to search on metadata key
  • helper to search on metadata value
  • helper to get metadata key and metadata value table field
  • helper to convert row's data to metadata object
• events
  • listener on new file
  • listener on edited file
  • on request, dispatch event on background job process
  • occ metadata:get

How to add metadata from your own app:

lib/AppInfo/Application.php

		$context->registerEventListener(MetadataLiveEvent::class, UpdateFilesMetadata::class);
		$context->registerEventListener(MetadataBackgroundEvent::class, UpdateFilesMetadata::class);

UpdateFilesMetadata.php

use OCP\EventDispatcher\Event;
use OCP\EventDispatcher\IEventListener;
use OCP\FilesMetadata\Event\MetadataBackgroundEvent;
use OCP\FilesMetadata\Event\MetadataLiveEvent;

class UpdateFilesMetadata implements IEventListener {
	public function __construct() {}

	public function handle(Event $event): void {
		if (!($event instanceof MetadataLiveEvent) &&
			!($event instanceof MetadataBackgroundEvent)) {
			return;
		}

		$node = $event->getNode();
		// only few files are interesting.
		if ($node->getMimetype() !== 'image/jpeg') {
			return;
		}

		// we only work on process initiated as background job, so ask for a re-run later
		if ($event instanceof MetadataLiveEvent) {
			$event->requestBackgroundJob();
			return;
		}

		$fd = $node->fopen('rb');
		if (false === $fd) {
			return;
		}

		try {
			$oldBufferSize = stream_set_chunk_size($fd, 1);
			$data = @exif_read_data($fd, 'ANY_TAG', true);
			stream_set_chunk_size($fd, $oldBufferSize);

                        // we have exif data in $data, let's index the photo taken date 
			$metadata = $event->getMetadata();
			$metadata->setInt('photo-taken', strtotime($data['EXIF']['DateTimeOriginal']), true);
		} catch (\Exception $e) {
		}
	}
}

How to use the (current) QueryHelper

		/** @var ConnectionAdapter $conn */
		$conn = \OCP\Server::get(ConnectionAdapter::class);
		$qb = $conn->getQueryBuilder();

		$qb->select('file.fileid', 'file.path')->from('filecache', 'file');

                // linking metadataQuery to existing query builder
		$metadataQuery = $this->filesMetadataManager->getMetadataQuery($qb, 'file', 'fileid');
                // add metadata to the request's results
		$metadataQuery->retrieveMetadata();
                // only returns result havin
		$metadataQuery->joinIndex('another');

		$expr = $qb->expr();
		$qb->where(
			$expr->lte(
				$metadataQuery->getMetadataValueIntField(),
				$qb->createNamedParameter(1, IQueryBuilder::PARAM_INT)
			)
		);
		$qb->orderBy($metadataQuery->getMetadataValueIntField(), 'desc');

[artonge]

JSON_THROW_ON_ERROR could be added for all JSON function calls.

[icewind1991]

Do I understand correctly that the metadata is only loaded on search requests atm?

I don't see any logic to trigger the loading for propfinds

[icewind1991]

does this need to be part of the public api?

[ArtificialOwl]

it might not be necessary

[ArtificialOwl]

Do I understand correctly that the metadata is only loaded on search requests atm?

GET request might wants to get their metadata, we could also imagine to use this new API to manage tags
and metadata will also be updated from iOS client (another PR is on its way)

I don't see any logic to trigger the loading for propfinds

If we don't load known indexed metadata, exceptions are thrown by https://github.com/nextcloud/3rdparty/blob/e6c45c6c0d4f92ffec621446fcc3b34584772b13/icewind/searchdav/src/DAV/SearchHandler.php#L102-L128

We could edit the code there to accept 'nc:metadata-*', but not sure this is something we want ?

[come-nc]

3000 lines /o\

[nickvergessen]

This PR has no version bump and therefore breaks all master setups because while the table is used, the migration to create it did not trigger.

[nickvergessen]

Follow up fix in #41351