[FIX] Doesn't work with self hosted Firecrawl

[azertylr]

Describe the bug

Hello,

I have a self-hosted version of Firecrawl, available at http://192.168.10.120:3002. Running
curl -X POST http://192.168.10.120:3002/v1/scrape \ -H 'Content-Type: application/json' \ -d '{ "url": "http://perdu.com", "formats" : ["markdown", "html"] }'
from the Khoj server docker is working fine. But when I want to use it with Khoj, it doesn't make a request to the firecrawl instance.

To Reproduce

Steps to reproduce the behavior:

• Setup a firecrawl instance
• Setup the API endpoint in Khoj
• make a request that need access to the internet.

Screenshots

Web scrapers

Server chat settings

Platform

• Server:
  • [X ] Self-Hosted Docker
• Client:
  • [X ] Web browser
• OS:
  • [ X] Linux: Ubuntu 24.04

If self-hosted

• Server Version : 1.36.4

Additional context

Add any other context about the problem here.

[azertylr]

I've restarted from scratch and it was working. I don't understand what happened

I've also tried the env variable and it works great :

- FIRECRAWL_API_KEY=
- FIRECRAWL_API_URL=

[debanjum]

Not sure. Setting up certain admin configs on Khoj requires service restarts. This maybe why your Firecrawl setup initially didn't get enabled?

Anyway good to know that Khoj works with self-hosted Firecrawl as I hadn't tried that yet.

Closing this issue for now. Feel free to re-open it if you feel something wasn't addressed

[azertylr]

It may be because I first setup my self hosted firecrawl using the admin UI and didn't restart.

Its good that using env var is working!