Can the pipe be more optimized for more speed?

[RawConvert]

I noticed that with the 5.0.0 appimage, DT is doing a lot of recalculation which seems potentially unnecessary.
When you change something late in the pipe, it is prone to recalculating the whole pipe, whereas you might hope it would realise it's only from the point of change that the pipe needs recalculating.
To demonstrate -
Take an image with some chunky processing so the effect is clearly visible, e.g. DorS hard deblur. (Or do -d perf)
Add something late in the pipe like Local Contrast or Tone Curve with a raster mask that is an instance of say Exposure.
Now toggle Local Contrast off and on.
Notice that the offs are quick but the ons are slow because of pipe recalc.
Yet neither the raster mask, nor anything else, has changed prior to Local Contrast.
If you're using Tone Curve to make some final tweaks, it is quite frustrating!
Ubuntu 22.04, no GPU card

[paperdigits]

Can you provide some logs that indicate this?

[RawConvert]

Is it not behaving this way for you?

[paperdigits]

I can not seem to reproduce.

[RawConvert]

Here are files showing a raw (CR2) with hard deblur in Diffuse/Sharpen and a Tone Curve instance with a raster mask. Switching off the Tone Curve is quick, switching is on is slow as DT re-calculates lots/all of the pipe.
Procedure -
0. save xmp with tone curve, DorS hard, exposure mask, as boatXmpA.txt

1. start dt -d perf. Lighttable comes up with no calculation details in the log, that seems ok.
2. doubleclick and the image appears quite slowly as DT calculates, ending at 28.3708 in the Terminal log.
3. switch off TC - this is quick, minimal processing, and the log has got to 153.8726
4. switch on TC - this is slow and the log is now at 226.3343
5. switch off TC - this is quick and the log has got to 285.3818
6. switch on TC - this is slow and the log is now at 370.9008
7. quit DT

Also note re. efficiency, though a relatively minor point, the histogram seems to be re-calculated unnecessarily... bring up any window on top of darktable, e.g. terminal, minimise it, and note this causes 4 extra lines in the terminal window, all re-drawing the histogram! At least on my system.

toneCurveTest-terminal-log.txt
boatXmpA.txt

The raw (CR2) is too big to upload, you can get it here -
https://discuss.pixls.us/t/storm-darragh-a-bit-breezy/46945

[jenshannoschwalm]

Yes - your rhetoric question "Can the pipe be more optimized for more speed" could be be answered with "YES - short pause - but how to do that safely?"

1. you define a mask in an exposure instance used later as a raster in Tone curve (TC)
2. Please note that your logs show that the pipe is processed from exposure if you toggle-on TC so the pixelpipe cache works at a good point, good and safe as the mask is generated here.
3. A sidenote, TC is somewhat even more complicated as it's internal histogram requires to be updated if anything happens at an earlier part in the pipe.
4. So for me this all looks good as it works as intended by current design.

The histo redrawing is a completely different story btw.

[jenshannoschwalm]

At least some significant issues with not required pipe processing have been fixed in master so closing this for now.
Please open a new issue if there is "something specific" that could/should be fixed. ok?