Using claudette for anthropic models shaves 2 seconds from start time.

[PiotrCzapla]

litellm eagerly loads most of it dependencies. Most of them are not used in ssage: IPython, openai, aiohttp, prompt_toolkit, jedi ... That adds almost 2 seconds to each run. I've switched lisette to claudette and ssage -h runs in 0.4 sec compared 2+ sec with Lissette.

I think it would make sense to selectively loading right library for the model family, as it could speed up ssage 5x.

Unfortunately, the fix isn't straightforward as claudette is not exactly drop in replacement for lissette. Would you be interested in PR ? If so should we try to back port features from lisette to claudette? Or would it make more sense to add alternative backends to lisette ?

The bits that are missing in claudette are:

• search - is missing from Chat
• max_steps, api_base, api_key - is missing from Chat.call
• Behavior of Chat.call with stream=True, differs; lisette - outputs a full result object; claudette - outputs partial response as string

importtime traces

lisette

$ time uv run python -X importtime -m shell_sage.core 2>importtime.txt         
uv run python -X importtime -m shell_sage.core 2> importtime.txt  1.30s user 0.38s system 66% cpu 2.505 total

$ grep import importtime.txt| awk '{print $5 " " $7; }' |sort -rn| grep -v '\.' | head
1679644 lisette
1368666 litellm
266972 IPython
210137 openai
105220 aiohttp
103790 fastlite
94338 httpx
94241 apswutils
50577 prompt_toolkit
48037 jedi

After changing to claudette it starts in 0.5 s

$ time uv run python -X importtime -m shell_sage.core 2>importtime-2.txt
uv run python -X importtime -m shell_sage.core 2> importtime-2.txt  0.34s user 0.11s system 95% cpu 0.471 total

$ grep import importtime-2.txt| awk '{print $5 " " $7; }' |sort -rn| grep -v '\.' | head
216720 claudette
213757 anthropic
64262 httpx
44634 fastlite
35389 apswutils
13245 markdown_it
11664 site
11306 pydantic
10805 anyio
9274 attr

[ncoop57]

Thanks @PiotrCzapla for looking into this speed! We actually use claudette and cosette (our openai equivalent) in shell_sage < 1.0.0 and so if you prefer to have the speed, I recommend use it instead. I switched to lisette because we've moved most of our development work to it and it cleaned up a lot of the code, so gonna go ahead and close this for now, but I'm happy to look into ways of speeding up shell sage by improving lisette or other areas that might need improving!

[PiotrCzapla]

I doubt that litellm will be open to a PR to reduce their load time as they target servers not cli and I don't think you can speed up lisette load without replacing litellm. So I guess we are stuck with additional 2 seconds per use.

[PiotrCzapla]

@ncoop57 I tackled the issue from a different angle: #63 -- feeling faster is as good as being faster in this case. If you find the added complexity acceptable, I would appreciate merging so I get it in solveit env :).