[Bug]: Prompt Editing Error when using Highres Fix

[troodpart]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

When using prompt editing, highres fix modifies the output of the image.
The "to" part of the edit seems to get applied to the "from" part disregarding the "when" condition during the upscale process. Even if "when" is set so only the "to" prompt should be used, the issue occurs.

Steps to reproduce the problem

1. Generate base image sequence using the following prompt. Use whatever model/vae/sampler you want.

Positive Prompt: [apple, : (orange,:1.3), : 0.99]
Negative prompt: (worst quality, low quality,:1.3)
Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 2576951082, Size: 512x512, Model hash: cca17b08da, Model: darkSushiMixMix_225D, VAE hash: 735e4c3a44, VAE: vae-ft-mse-840000-ema-pruned.safetensors, Clip skip: 2, Script: X/Y/Z plot, X Type: Prompt S/R, X Values: "0.99, 0.95, 0.1", Version: v1.6.0-2-g4afaaf8a

2. Generate sequence again, this time upscaling using the following settings. Different upscalers produce the same effect.

Hires upscale: 2, Hires upscaler: R-ESRGAN 4x+ Anime6B, Denoising strength: 0.5,

3. Compare outputs - note how your apples get orange'd earlier in the upscaled image

sysinfo-2023-11-12-18-34.txt

What should have happened?

Highres fix should just upscale the image, not modify its contents.

Sysinfo

sysinfo-2023-11-12-18-34.txt

What browsers do you use to access the UI ?

Mozilla Firefox

Console logs

venv "C:\Users\Oosko\Pictures\ai\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.6.0-2-g4afaaf8a
Commit hash: 4afaaf8a020c1df457bcf7250cb1c7f609699fa7
Launching Web UI with arguments: --no-half-vae --xformers
*** "Disable all extensions" option was set, will not load any extensions ***
Loading weights [cca17b08da] from C:\Users\Oosko\Pictures\ai\stable-diffusion-webui\models\Stable-diffusion\darkSushiMixMix_225D.safetensors
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Creating model from config: C:\Users\Oosko\Pictures\ai\stable-diffusion-webui\configs\v1-inference.yaml
Startup time: 11.3s (prepare environment: 2.2s, import torch: 4.8s, import gradio: 0.7s, setup paths: 0.7s, initialize shared: 0.2s, other imports: 0.5s, setup codeformer: 0.1s, scripts list_optimizers: 1.2s, create ui: 0.3s, gradio launch: 0.5s).
Loading VAE weights specified in settings: C:\Users\Oosko\Pictures\ai\stable-diffusion-webui\models\VAE\vae-ft-mse-840000-ema-pruned.safetensors
Applying attention optimization: xformers... done.
Model loaded in 2.4s (load weights from disk: 0.7s, create model: 0.3s, apply weights to model: 0.9s, load VAE: 0.2s, calculate empty prompt: 0.1s).
X/Y/Z plot will create 3 images on 1 3x1 grid. (Total steps to process: 60)
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 11.30it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.08it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.11it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 60/60 [00:05<00:00, 11.73it/s]
X/Y/Z plot will create 3 images on 1 3x1 grid. (Total steps to process: 120)███████████| 60/60 [00:05<00:00, 14.95it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 13.82it/s]
Total prTile 1/9 16%|██████████▎                                                      | 19/120 [00:01<00:06, 14.89it/s]
        Tile 2/9
        Tile 3/9
        Tile 4/9
        Tile 5/9
        Tile 6/9
        Tile 7/9
        Tile 8/9
        Tile 9/9
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:06<00:00,  3.30it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 14.95it/s]
Total prTile 1/9 49%|███████████████████████████████▉                                 | 59/120 [00:11<00:04, 12.59it/s]
        Tile 2/9
        Tile 3/9
        Tile 4/9
        Tile 5/9
        Tile 6/9
        Tile 7/9
        Tile 8/9
        Tile 9/9
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:06<00:00,  3.28it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.12it/s]
Total prTile 1/9 82%|█████████████████████████████████████████████████████▋           | 99/120 [00:21<00:01, 12.70it/s]
        Tile 2/9
        Tile 3/9
        Tile 4/9
        Tile 5/9
        Tile 6/9
        Tile 7/9
        Tile 8/9
        Tile 9/9
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:06<00:00,  3.28it/s]
Total progress: 100%|████████████████████████████████████████████████████████████████| 120/120 [00:29<00:00,  4.04it/s]
Total progress: 100%|████████████████████████████████████████████████████████████████| 120/120 [00:29<00:00,  3.28it/s]

Additional information

This wasn't an issue on earlier builds - even the ones after the highres fix system was updated. I think a commit in the past 3-4 months caused the change. I looked back at some older images and it seems the ones in August didn't have the problem, but the ones later in September did. I probably git-pulled updates at some point between them, but I don't have a record of exactly when or what build.
I don't think this issue is extension related since I can make it happen with all of mine disabled.

[catboxanon]

Does enabling the Use old prompt editing timelines under Compatibility in settings fix this? You say this functionality changed sometime between August and September which aligns with when that PR was merged. #12457
I think that wiki page needs to be updated to make this slight change in functionality more clear. It's documented in https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Seed-breaking-changes but not the features page.

[troodpart]

Does enabling the Use old prompt editing timelines under Compatibility in settings fix this? You say this functionality changed sometime between August and September which aligns with when that PR was merged. #12457 I think that wiki page needs to be updated to make this slight change in functionality more clear. It's documented in https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Seed-breaking-changes but not the features page.

Yeah that fixed it. Missed that bit when I was hunting through the settings. Thanks!

[lichdkimba]

Does enabling the Use old prompt editing timelines under Compatibility in settings fix this? You say this functionality changed sometime between August and September which aligns with when that PR was merged. #12457 I think that wiki page needs to be updated to make this slight change in functionality more clear. It's documented in https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Seed-breaking-changes but not the features page.

Yeah that fixed it. Missed that bit when I was hunting through the settings. Thanks!

hello, are you sure it is fixed after enabling this? I enable it but likely the upscaler always the "from" part and the result it different compared to not using the Highres Fix.

[lichdkimba]

set Denoising strength of hires fix low to solve, but i think the prompts used by Hires fix may still not align with the inital used one.