[Feature request] Manually update xmp files for selected collections

[macalje]

Problem:
I'm always frustrated when I start darktable (DT) and have look for updated XMP files on startup turned on which make the start up of DT to take a very very long time. In my case, I want that on part of the year when we are working on the familys year books, a work including DT when we are selecting images which is done on 3 computers (both windows and linux). For this we are only using the lighttable mode. For editing, we are also using DT on these 3 computers, but there it's not a problem because we complete every photoediting work on the same computer.

Solution:
In collections, right click the collection you would like to check for updated xmp files. In the right click menu you get the option of Check for updated XMP, Update XMP or something similar.

When the option for checking for and update the XMP files have been selected:

• A window (like the fields that is used for the modules in darkroom) appears in the collection window below the selected collection. This approach open up the possibility to add more options with different functionality in the future, if someone finds a need for it. (Like the possibility to select if DT only should check for possible updates of the XMP files or if there is the need to integrate the existing function when DT is checking for updates at start up as it is today; where you get the choice to choose which one to keep. But this suggestion only is about updating XMP files.)
• At the top of the window DT informs the user how many images that is affected. This value is a counter collected from DT "looking" in the selected collection. It could look like something like this: Images that will be checked for updated XMP files: N or something like that.
• If a certain number of images is found to be affected and there is reason to belive this will take a "long time" (I have no suggestion of what would be considered a long time), the report of image affected by the request would be followed by a warning in red that the action to update the XMP files might take some time. This could look something like this: There is the possibility this action might take a long time to execute.
• This window shows a (bright green against black background, I personally love that look:)) taskbar that shows progress, where DT has calculated how many images that is affected by the request and adjusted the scale of the taskbar steps so that every step in the taskbar is equal to one image processed. So if there is only 2 images in the selected collection, the taskbar will only take 2 steps, if there is 100 images in the selected collection, the taskbar will take 100 steps.
• A counter display (would be great to have bright green against black background again:)) which shows the percentage in integers where 0 percent is equal to 0 photos checked, 1 percent is equal to at least 1 percent or more of the total amount of images that is about to be checked and 100 is equal to all the images affected by the request is checked and updated. This means that 100 percent only show up when the process is completely finished.

When finished, at the bottom of the window:

• There is a report telling the user how many pictures that was actually checked. This number is collected from a counter counting every image that has been checked: Number of checked images: n
• There is a report telling the user if n equals N: if n = N then 'All images in the collection have been checked.' in bright green color (the function has done exactly what the user wanted it to do). elif n < N then 'Darktable was not able to check all images in the collection for updated XMP files. in yellow (this isn't necessary a problem but needs to be flagged anyway and my belive is that yellow text would be enough). elif n > N then 'Darktable has checked more images than exists in the collection for updated XMP files.' in red color (systems that do more than you ask them to do is problematic and when that happens the user needs to be warned).
• There is a report telling the user how many XMP files that was replaced with updated XMP files: "X XMP files was updated." where X is based on a counter counting every XMP file that was updated.

Alternative
Instead of right click a collection this could also be like an own module called something like collections XMP or similar. Then it could work in that way that the module check for updated XMP files like described above for all images that is selected in Lighttable. This would give a more stringent "feel" compared to how DT looks and works for the user. The user then can use collections and the associated filters as of today to select all images the user want to check for updated XMP files, then select all or only the pictures of interest for the user in Lightable, go to this "module" and run it on selectet/marked images. (I do apologise for several updates of this post, my kid conmtroled the mouse "on my behalf" so instead of preview it became update comment.)

The module like window described above can also be a floating window if that's more convenient.

The taskbar and percentage display doesn't need to be in the suggested colors from above. But I love bright green against black in contexts like this.

[difrkaguilar]

The module like window described above can also be a floating window if that's more convenient.

Can you check #12828 and look if part of this post is related with your FR please?

[macalje]

I would say that #12828 is a great example of feature that could be inclluded as one of the functionalities in the Lighttable XMP update "module". See my mentions about future functionalities that I might not have thought about in my description and motivation for a windows or module like field.

[macalje]

The module like window described above can also be a floating window if that's more convenient.

Can you check #12828 and look if part of this post is related with your FR please?

If the question also applies to if I with floating window mean a window like the one that #12828 refers to, the answer is yes.

[macalje]

I'm always frustrated when I start darktable (DT) and have look for updated XMP files on startup turned on which make the start up of DT to take a very very long time.

What about the option in a module to look for updated xmp with a button click (basically upon request)?

Thats exactly what I'm asking for but with some more information/features that also give me information about the progress in updating the xmp:s AND most important of all, the ability to select which pictures XMP files I want DT to update if there is an updated XMP file available.

I think "why do I want the progress information?" is a fair question: Well, on both my main windows and my Linux machine DT sometimes stop "responding". I've learned that most of the time, it hasn't crashed but I've asked it to handle to many pictures. So if I just let it be for a couple of minutes or hours it usually "comes back". But sometimes it doesn't. In these cases it usually have done what I asked it to do but just didn't make it "back" from not responding. In these cases it would be nice to have continues feedback of the progress because then I would be able to see that it's working with my "too large question" and when it says 100%, well, if it's not back and running I can just terminate DT and restart. (DT is really good at handling this which I guess is thanks to the XMP files.)

A scaled progress bar as described above would also give the user some indication of how long time the operation might take. I'm not asking for DT to make an estimation but I want to give the user (in this case me:)) an indication about how long it might be before all the requested pictures XMP files is checked for potential updates.

[macalje]

I think the progress report might be overly complicated for the little benefit it might have. Having a "working..." Is more than enough in most cases like other dt modules.

Well. In most other DT modules there's only one image that is being processed and you have quite good capability to verify the result. Another example on where DT already have some sort of counter involved is when you export multiple files. Then there is a small report/feedback telling the user how many images that has been exported by the total number of image that is requested by the user to be exported. Even in this case it will be mainly easy to verify the result since the images will appear somewhere where the user have told DT to put them. If they're not there, then something haven't worked as expected. In this case however, you can have thousands of images you want to have checked for potential updates and it might be quite overwhelmingly to verify if everything has worked as expected. My suggestions is not idiot proof and there will be plenty of room for users do do "logical bugs/mistakes" in how they use my suggested function without my suggested feedback/reports will help the user to notice there mistake. But it is something that at least will give, as I see it, the most basic suport in verifying the execution.

What about the taskbar then. My wish for that one comes down to that I would really appreciate the ability to make a some what qualified guess or estimate for how long the initiated process seems to take. To get that this way, you will make that estimate taking into account the computer's total load. So if you're running other apps on the computer that affects how fast DT can work, that would show up because the next step on the taskbar is only done when the image is checked, which will take different time depending on the computers total work load. In that way you will have a quite good indicator that will help you determin if you should/can wait or if you should/can go and do something else (like sleep because it's night... :)) while you're waiting which is sensitive for total work loads. And as I mentioned above, if DT stops responding but the taskbar still moves then it's not responding because it's occupied. If it's not responding and the taskbar is stuck on the same place then it's probably (but not necessary) crashed.

This is, though, my thoughts and what I belive would suite my needs. Thats the best I can do in this wish for this feature. Since I can't do the code myself I'm left to the opinions and judgments of those who may want to take care of this request/desire and actually can program the function might do. :)

[github-actions]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[quazardous]

I'm using a NAS as main location.

"Local copies" is a big features with NAS. So thx for that !

but I miss the possibility to update a selection of images (ie when I switch between computers).

Why not just add a new action "check XMP" somewhere in the "selected image" panel ?

Please

[wpferguson]

Preferences->Storage->look for updated XMP files on startup

[quazardous]

Preferences->Storage->look for updated XMP files on startup

Not possible with a NAS and hundreds of Go: too slow (with my home lan at least).

Or DT should allow to maintain some light "touch" repo/sqlite on the NAS_or_at_specific_path to speedup things (working with NAS or slow storage could be improved but it's not in the scope of this issue).

So just allow to refresh the current selection is not so bad.

Without touching the UI, it could be part of the local files sync with a big sync_the_stuff (but there will always be a case when doing all together will break something).

[ptilopteri]

or you could make sure your "NAS" has the same address on all your work stations, then each of your photos would have the same latest, unique xmp

[quazardous]

or you could make sure your "NAS" has the same address on all your work stations, then each of your photos would have the same latest, unique xmp

Sorry I don't understand

Xmp is on NAS ok

the PB is how to discover modifications of XMP done by other computers with DT without scanning 500 Go of photos over network.

Is the XMP somewhat address dependant ?

But in the end I know which photos I have worked on even if it's on my travel laptop (in the typical workflow it's the last subfolders). I just need to refresh the XMP on my tower PC at some point.

[ptilopteri]

if you have configured xmp updates while working photo and your photos are on a "NAS", the xmp files on the "NAS" are current.
if you locally copy the files to bypass delays accessing "NAS", simply "resync local copy" after you have worked the "local copied files" and the "NAS" will reflect the current updated xmp files.
you are trying to use a sledge hammer where all you need is a tack hammer.

[quazardous]

Ok I understand and I use it like it. Thx for your kind explanations.

But that does not answer the need to update DT local DB from XMP in a multiple working computers environnement without having to scan the whole NAS at startup.

[ptilopteri]

put the db on the "NAS"

[quazardous]

This use case was discussed elsewhere but seams dangerous to me.

When you work on local files on a laptop on the go (NAS not available) you need a DB. Merging multiple DB or working with multiple DT installation seams risky or at least not dumb proof. And each DT instance locks the DB at startuo. So if you forget to close DT on one computer you're screwed. As for today this is bad idea. Maybe one day we will see a DT server for this kind of use case.

[macalje]

Still, some kind of module or functionality to "manually" trigger a check for updated xmp:s on only a selected amount of pictures would be of great help. Especially when working with the same pictures on different computers, where the pictures are stored on a NAS or in a cloud service.

[parafin]

Well, how long does it actually take to scan all XMPs over network? DT doesn't (shouldn't) read the actual big image files during this process.

[quazardous]

Well, how long does it actually take to scan all XMPs over network? DT doesn't (shouldn't) read the actual big image files during this process.

It's not relevant because it will always increase. Scanning all files at startup is bad by design. It's just a quick workaround because of NAS users. But a real NAS workflow is yet to be found (A DT server).

For me it takes 3 minutes at least before splash screen to appear (5k photos increasing).

[github-actions]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[ralfbrown]

#17202 will provide an updating progress percentage while darktable checks for changed sidecars at startup.

[github-actions]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[aa956]

Still very relevant.

Startup time for a darktable 4.8.1 is 10+ minutes if enabled [x] look for updated XMP files on startup.

Multiple minutes for darktable to show the main window after starting is a bug in any case, even if I have a slow storage I'd expect application to start and show some kind of user interface - progress bar perhaps or some way to cancel the scan and start using the software?
Should it be filed separately?

Use case:

Images are on a NAS samba share on a 1GbE LAN.
Images: ca 130000 files.
2 computers accesing the images (desktop and laptop).

Have not yet tried to run a darktable on a WiFi or VPN link as this would probably take days.

darktable 4.8.1 flatpak install on debian 12:

aa956@desktop02:~$ flatpak list --app
Name                      Application ID                           Version         Branch         Installation
darktable                 org.darktable.Darktable                  4.8.1           stable         user
digiKam                   org.kde.digikam                          8.5.0           stable         user
Krita                     org.kde.krita                            5.2.6           stable         user
aa956@desktop02:~$ 

[aa956]

put the db on the "NAS"

That may be a good idea but I immediately see 2 problems:

1. Images directory may be mounted in different locations, e.g. /home/username/mnt/fileserver/media/camera/ for desktop and /media/username/... on a laptop. Throw some Windows machines in a mix and it may become a combination of H:\camera and \\fileserver.home.lan\media\camera.
2. I'd like to use XMP sidecars to share data between digikam and darktable and some custom scripts (e.g. vision AI model captioning is getting better every day now and its usage from python is quite simple).
3. NAS is slow (1 GbE ethernet in best case, HDD-s probably for a few more years). 10+ GbE and 20+ Tb SSD storage is clearly out of budget for a lot of home users.
4. Remote access: accessing a sqlite database file through slow (both latency and throughput) unreliable VPN link is a sure way to corrupt a database.

Edit: added points 3-4

[quazardous]

put the db on the "NAS"

I've tried and it's not "safe".

IMO The only way would be to create a server version of dartktable (could be the same binary with --server on the command line).

The clients darktable could communicate with the server running on the NAS to query distant files and db.