LibreChat frequently unable to display images generated by DALL-E

[lindner-tj]

What happened?

LibreChat v0.7.4-rc1 running via Docker Compose on an Ubuntu 22.04.4 LTS VM

Hello All,

I am unable to get LibreChat to consistently display images generated using DALL-E. I send prompts like "draw a cute cat" to an OpenAI GPT Model to make it generate DALL-E images using the DALL-E plugins from the plugin store. It sometimes succesfully displays the image, but a lot of times it does not. Changing the GPT or DALL-E version doesn't seem to have an effect.

When it works, the image gets displayed just fine, when it doesn't the DALL-E prompt generated by GPT shows in the browser followed by a "generated image" line:

There seems to be no pattern to this behavior. Using tcpdump from the api service's container I'm certain I can rule out firewall or proxy blocks to any of the OpenAI remote IP adresses.

When the error occurs, the docker log for the api container always shows the following lines, indicating an issue with correctly saving the image file:

2024-07-10 14:28:40 error: [saveFileFromURL] Error while saving the file: Cannot read properties of undefined (reading 'status')
[tool/end] [1:chain:AgentExecutor > 3:tool:DALLE3] [20.09s] Exiting Tool run with output: "![generated image](/images/664f16d57f38d0edd47df187/img-fec228b2-2281-447d-936d-6008316b5e05.png)"

in the .env file the following variables are set using the same key:

OPENAI_API_KEY=<myKey>
DALLE_API_KEY=<myKey>
DALLE3_API_KEY=<myKey>
DALLE2_API_KEY=<myKey>

Any ideas what's causing this?

Kind regards, Tom

Steps to Reproduce

1. Download and select a DALL-E plugin from the plugin store
2. Use any OpenAI GPT model to make a DALL-E Prompt
3. maybe LibreChat will display the image successfully, maybe it won't
4. repeat

What browsers are you seeing the problem on?

Microsoft Edge

Relevant log output

2024-07-10 14:28:40 error: [saveFileFromURL] Error while saving the file: Cannot read properties of undefined (reading 'status')
[tool/end] [1:chain:AgentExecutor > 3:tool:DALLE3] [20.09s] Exiting Tool run with output: "![generated image](/images/664f16d57f38d0edd47df187/img-fec228b2-2281-447d-936d-6008316b5e05.png)"

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

I moved this to discussion because it can't be reproduced.

This could depend on several things, including your docker setup.

Are you using the default docker setup?

[vislamu]

This is my screenshot, which shows the following when I request to generate two different rose images using the relevant settings shown in the picture. I have retrieved your latest code. Could you please help me identify the issue. I installed it locally using npm on a Mac device

[lindner-tj]

Hi @danny-avila

thank you for your response. Indeed, I am not running a default docker setup on my server. I have since done some further testing. Both on my company laptop (Windows 11) and on my private machine (Linux Mint). In both cases with the absolute minimum and most default setup possible. And on both machines I observe the same error as on the server. With a few differences: running Librechat on localhost using Docker Desktop I get no log entries when it fails to display the image. This ist true on both PCs.
On my private machine it fails, too. However, at a much lower rate. Maybe 5 % of all requests fail to show the image. The work laptop is very similar to my server in terms of error rate.

I think the only relevant difference between my work laptop and my private one is the much simpler network setup for the latter.

Is it possible that a part of LibreChat is super sensitive to timing and doesn't wait long enough before it looks for the image? Maybe it just hasn't finished downloading yet or it is still being checked by anti-malware.

[luke3butler]

Having the same issue. It hasn't been functioning for some time now, but definitely was at one point.

Here is the image of a majestic horse standing in a lush green meadow. The horse has a shiny brown coat with a black mane and tail, set against a background of rolling hills and a clear blue sky with a few fluffy white clouds. The scene is bright and vibrant, capturing the beauty and grace of the horse in its natural habitat.

![A Majestic Horse](https://images/65a3413359ad8f6d967aba4b/img-56e16166-7769-45bb-b94e-a1c1c004fa61.png)

Manually editing the message, adding the domain, and saving it will result in the image displaying properly.

I assumed there was a config change and I needed to add a base URL or something, but haven't seen that anywhere in the docs/code so far.

[luke3butler]

I also just noticed that if I remove https:/ the image is displayed, and all followup images will not require this manual edit.

[pechyonkin]

@luke3butler I am facing the same issue, can you tell me what you mean by:

Manually editing the message, adding the domain, and saving it will result in the image displaying properly.

How do you do that?

[pechyonkin]

I was able to resolve the issue. It was my firewall blocking downloads from domain windows.net. After I allowed that domain, the images started appearing.

[lindner-tj]

Hi @pechyonkin, did your firewall block every image or just some?

[pechyonkin]

@lindner-tj my firewall was blocking everything from windows.net. After whitelisting the issue was fixed completely for me. I hope this helps.

[danny-avila]

thanks for the update @pechyonkin

[lindner-tj]

@pechyonkin whitelisting said domain was pretty much the first thing I did. It unfortunately didn't improve the situation. The issue persists, but there are no more dropped connections, both in the firewall as well as proxy. I set up a fresh instance with default config on my local machine (work laptop, windows 11) and for some reason images now work flawlessly. The server still has the issue though, even on the newest release.

[luke3butler]

I'm using this as part of my prompt prefix, and now image URLs have been correct every time. It can probably be optimized a bit.

When generating images with a link, do not include "https://" or "http://", but DO prefix generated image urls with an initial "/".

@danny-avila Would something along these lines fit within imageInstructions? Maybe with an adjustment for handling/detecting both internal and external URLs.

[danny-avila]

Sure but the instructions will be rendered unnecessary with the coming update: #3607

Feel free to submit a PR to improve the current implementation!

[lindner-tj]

I found the cause for my issue: It was the firewall, but in a really advanced and non-obvious way. Using tcpdump, I saw that the DALL-E images all seem to come from a certain URL. I've whitelisted this before, but now adding this URL to absolutely all possible whitelists and exceptions makes image display reliable enough for me. It now works in 9/10 cases. I am going to close this discussion.

[chirudeep37]

Can you elobarate more on how you fixed it. I got the same issue.

[ringhofd]

For us dns-resolution of theImageUrl (oaidalleapiprodscus.blob.core.windows.net) stopped working in our company network. I suppose this is some kind of automatic malware/abuse block applied by the dns provider/infra we use.

[chirudeep37]

sorry to bother brother but I'm not well versed with network technicalities but familiar with ubuntu and docker, I'm running it locally and getting the same error as mentioned in the thread. can you explain in simple words?

[ringhofd]

Check if oaidalleapiprodscus.blob.core.windows.netdoes resolve: Open a terminal and use nslookup:

nslookup oaidalleapiprodscus.blob.core.windows.net

[chirudeep37]

This is what I got
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
oaidalleapiprodscus.blob.core.windows.net canonical name = oaidalleapiprodscus.privatelink.blob.core.windows.net.
oaidalleapiprodscus.privatelink.blob.core.windows.net canonical name = blob.sat10prdstf01a.store.core.windows.net.
Name: blob.sat10prdstf01a.store.core.windows.net
Address: 20.150.70.100
Name: blob.sat10prdstf01a.store.core.windows.net
Address: 64:ff9b::1496:4664

[ringhofd]

I see this behavior too now. In the debug logs nothing really shows up. The image files are not saved in the client/public/images folder.
firewall logs also do not show any blocks or drops. how can I triage this?

[danny-avila]

Do you have mounted volumes? in the default compose file, this is what persists the images:

    volumes:
      - type: bind
        source: ./.env
        target: /app/.env
      - ./images:/app/client/public/images
      - ./logs:/app/api/logs

[ringhofd]

Exactly I have a volumemount in k8s at /app/client/public/images - this is backed by a pvc.
The mount looks fine and I even see older images I generated with dall-e in this folder and it's subfolders.
It must have broken recently.

volumeMounts:
            - name: librechat-data
              mountPath: /app/client/public/images
              subPath: images
volumes:
        - name:  librechat-data
          persistentVolumeClaim:
            claimName: librechat

[ringhofd]

Apparently the hostname for theImageUrl (DALLE3.js) does not resolve in our local network. I have to go investigate/fix this with our admins.