Unable to update properties of Image

[jeff-j-chen]

Describe the bug

Certain properties of a gr.Image cannot be updated with the gr.update() function. These include parameters like the tool and shape of an Image, which should be able to be changed: update is supposed to "[take] as parameters any of the constructor parameters for that component."

Have you searched existing issues? 🔎

• I have searched and found no existing issues

Reproduction

import gradio as gr
def welcome(image):
    return gr.update(shape=(100,100), tool="sketch")

with gr.Blocks() as demo:
            image = gr.Image(label="Background", source='upload', type="pil")
            image.change(welcome, image, image)

demo.queue().launch(share=True)

Logs

Traceback (most recent call last):
  File "/home/jeff/SSD_2/ocv/lib/python3.10/site-packages/gradio/routes.py", line 442, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/jeff/SSD_2/ocv/lib/python3.10/site-packages/gradio/blocks.py", line 1392, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "/home/jeff/SSD_2/ocv/lib/python3.10/site-packages/gradio/blocks.py", line 1317, in postprocess_data
    prediction_value = postprocess_update_dict(
  File "/home/jeff/SSD_2/ocv/lib/python3.10/site-packages/gradio/blocks.py", line 448, in postprocess_update_dict
    update_dict = block.get_specific_update(update_dict)
  File "/home/jeff/SSD_2/ocv/lib/python3.10/site-packages/gradio/blocks.py", line 321, in get_specific_update
    specific_update = cls.update(**generic_update)
TypeError: Image.update() got an unexpected keyword argument 'shape'

### System Info
```shell
Gradio Environment Information:
Operating System:  Linux


gradio version:  3.38.0

gradio Dependencies:
  gradio-client: 0.2.10
  semantic-version: 2.10.0
  typing-extensions: 4.7.1
  jinja2: 3.1.2
  requests: 2.27.1
  pydantic: 1.10.2
  aiofiles: 23.1.0
  orjson: 3.9.2
  pillow: 9.5.0
  pyyaml: 6.0
  numpy: 1.24.2
  httpx: 0.24.1
  markupsafe: 2.1.1
  ffmpy: 0.3.1
  uvicorn: 0.23.1
  markdown-it-py: 2.2.0
  websockets: 11.0.3
  packaging: 22.0
  pandas: 2.0.3
  matplotlib: 3.5.1
  aiohttp: 3.8.3
  fastapi: 0.100.0
  huggingface-hub: 0.16.4
  mdit-py-plugins: 0.3.3
  pydub: 0.25.1
  altair: 5.0.1
  python-multipart: 0.0.6


gradio_client version:  0.2.10

gradio_client Dependencies:
  websockets: 11.0.3
  fsspec: 2023.6.0
  packaging: 22.0
  typing-extensions: 4.7.1
  huggingface-hub: 0.16.4
  httpx: 0.24.1
  requests: 2.27.1

Severity

I can work around it

[freddyaboulton]

Hi @4a454646 ! Sorry about the confusion but we intentionally don't support updating the shape of gr.Image. @pngwn @aliabid94 Is there a reason for that?

[jeff-j-chen]

Hi @4a454646 ! Sorry about the confusion but we intentionally don't support updating the shape of gr.Image. @pngwn @aliabid94 Is there a reason for that?

Interesting, it would be great to have that clarified on the Gradio documentation. I have been on an absolute journey trying to get this functionality, and discovered several bugs in the process.

My first thought after trying that was to instead manually resize the image with pillow. So I tried that, using the following code:

import gradio as gr
def welcome(image):
    return image.resize((100, 100))
with gr.Blocks() as demo:
    image = gr.Image(label="Background", source='upload', type="pil")
    image.upload(welcome, image, image)
demo.queue().launch(share=True)

And that actually works for a regular image. HOWEVER, my goal is to resize a sketch image. Once I add the parameter tool="sketch" to gr.Image, I get the following error:

Traceback (most recent call last):
  File "/home/jeff/SSD_2/ocv/lib/python3.10/site-packages/gradio/routes.py", line 442, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/jeff/SSD_2/ocv/lib/python3.10/site-packages/gradio/blocks.py", line 1387, in process_api
    inputs = self.preprocess_data(fn_index, inputs, state)
  File "/home/jeff/SSD_2/ocv/lib/python3.10/site-packages/gradio/blocks.py", line 1230, in preprocess_data
    processed_input.append(block.preprocess(inputs[i]))
  File "/home/jeff/SSD_2/ocv/lib/python3.10/site-packages/gradio/components/image.py", line 278, in preprocess
    mask_im = processing_utils.decode_base64_to_image(mask)
  File "/home/jeff/SSD_2/ocv/lib/python3.10/site-packages/gradio/processing_utils.py", line 58, in decode_base64_to_image
    image_encoded = extract_base64_data(encoding)
  File "/home/jeff/SSD_2/ocv/lib/python3.10/site-packages/gradio/processing_utils.py", line 49, in extract_base64_data
    return x.rsplit(",", 1)[-1]

And of course I can't modify tool using an update function, as we've seen above. I tried modifying the function to instead simply resize the image, rather than returning a gr.update call: but no luck, the same error arises, or the image is unchanged.

My next idea was to have two image boxes, one for the upload, and one for the resized image to sketch on. Using the upload function, I would simply transfer the value over.

import gradio as gr
def welcome(image):
    return gr.update(value=image.resize((100, 100)))
with gr.Blocks() as demo:
    image = gr.Image(label="Background", source='upload', type="pil")
    image_sketch = gr.Image(label="SKETCH HERE", source='upload', type="pil", tool="sketch")
    image.upload(welcome, image, image_sketch)
demo.queue().launch(share=True)

However, this breaks something about image_sketch: for one, you become unable to upload anything to it, as the box simply breaks:

And this functionality breaks no matter what the callback function is. Using literally a worthless function:

def welcome(image):
    pass

has the same result, so long as image.upload has an output set as image_sketch. This is an interesting bug, but not a huge deal, except for the fact it also breaks the any sketching functionality. Upon uploading any image to image, it is indeed resized and sent over the image_sketch, but there is no option to sketch. The only button left is a download button.

By this point I was at my wit's end, so I tried the easiest thing: setting the shape of image upon creation, but I can't reference image while defining itself.

I have wasted an entire day trying to figure this out. My only goal is to downscale the image proportionally after upload, so that I can sketch on it without lagging. sketch mode is very laggy, even on powerful systems, for large images. If I set a hard limit—for example, shape=(1024, 1024), any non-square image will be cropped, which removes any aspect ratio.

[abidlabs]

Hi @4a454646 this should now be possible on main! You can return a gr.Image() component from your function to update any of its properties.

I've modified your original example a little bit:

import gradio as gr

def welcome():
    return gr.Image(shape=(100,100), tool="sketch")

with gr.Blocks() as demo:
            image = gr.Image(label="Background", source='upload', type="pil", interactive=True)
            image.upload(welcome, None, image)

demo.queue().launch()