How to keep the background and only change the object been segmented?

[imadcat]

When I used the semantic segmentation model to change the specific object in the foreground, I would like to keep the background pixels completely unchanged.
Is there any prompt tricks to achieve that? Or it is not supported by the current algorithm?

Note this is almost the opposite question to this one: #123 (comment)

[geroldmeisinger]

Inpainting or Photoshop

[imadcat]

Yes I know the concept of inpainting, the question is whether ControlNet is capable of doing so?

[icemusike]

I have the same question, how to only change background but keep model in place.

[geroldmeisinger]

you can use inpainting with control net

[nighting0le01]

can we use 4 channel unet + inpainting to do so?