Prompt times don't match the real times in a screen recording, and zsh4humans seem to be getting quite a good advantage

[ericbn]

It's hard to be sure how exactly zsh-bench is measuring what it claims to be measuring, by looking at its code.

I'm going to trust a screen recording as the source of truth for the prompt times in my machine. I've recorded zim and zsh4humans, 3 times each. ​Each time the steps are:

1. Sleep 1 second. In this time I type exit and ENTER, so it's used as input in the next steps.
2. Print the framework name. I consider the counting starts when the output of this print appears in the recording.
3. Start a new shell with HOME as the specific framework installation dir. I consider the counting ends when the prompt appears.

This translates to: for f in zim zsh4humans; do repeat 3 do; sleep 1; print ${f}; HOME=${PWD:h}/${f} zsh -li; done; done

The steps above are executed in a dir with a git repo with 1,000 directories and 10,000 files, set up using the code here and here.

Each framework was installed with the setup script here and here, respectively for zim and zsh4humans.

This should be enough to guarantee that the recordings are using the same scenario used by zsh-bench.

This is the screen recording:

This is the times extracted from checking the recording frames:

	begin_frame	prompt_frame	frames	ms
zim 1	161	169	9	300.000
zim 2	200	207	8	266.667
zim 3	238	246	9	300.000
zsh4humans 1	276	282	7	233.333
zsh4humans 2	316	321	6	200.000
zsh4humans 3	354	359	6	200.000

The recording has 30fps, so each frame represents 33.333ms. We can consider the error of each measurement above to be +/- 33.333ms.

This is what I get when running zsh-bench on my same machine. Ran 3 times each of the mentioned frameworks:

❯ ./zsh-bench zim
==> setting up a container for benchmarking ...
==> benchmarking zim ...
creates_tty=0
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=115.790
first_command_lag_ms=149.910
command_lag_ms=62.535
input_lag_ms=28.830
exit_time_ms=47.530

❯ ./zsh-bench zim
==> setting up a container for benchmarking ...
==> benchmarking zim ...
creates_tty=0
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=196.126
first_command_lag_ms=229.765
command_lag_ms=142.110
input_lag_ms=29.402
exit_time_ms=48.146

❯ ./zsh-bench zim
==> setting up a container for benchmarking ...
==> benchmarking zim ...
creates_tty=0
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=196.878
first_command_lag_ms=231.291
command_lag_ms=140.756
input_lag_ms=27.556
exit_time_ms=48.560

❯ ./zsh-bench zsh4humans
==> setting up a container for benchmarking ...
==> benchmarking zsh4humans ...
creates_tty=1
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=30.361
first_command_lag_ms=100.323
command_lag_ms=4.997
input_lag_ms=14.536
exit_time_ms=10.895

❯ ./zsh-bench zsh4humans
==> setting up a container for benchmarking ...
==> benchmarking zsh4humans ...
creates_tty=1
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=30.458
first_command_lag_ms=101.612
command_lag_ms=5.198
input_lag_ms=12.339
exit_time_ms=11.051

❯ ./zsh-bench zsh4humans
==> setting up a container for benchmarking ...
==> benchmarking zsh4humans ...
creates_tty=1
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=29.819
first_command_lag_ms=108.122
command_lag_ms=5.123
input_lag_ms=13.146
exit_time_ms=11.05

What is odd

zim first_prompt_lag_ms times in zsh-bench fluctuate quite a lot. It's an average of 169.598ms with stdev of 46.601! In the recordings the average was 288.889ms with a stdev of 19.245, which was actually a stdev of 0.577 in terms of frames (it's just one frame more in one recordings). zsh-bench should be more precise than a 30fps recording.

Also from zsh-bench's code looks like this output is the min value. It would be good to know what is the stdev of values inside a zsh-bench run, to make sure the values are not fluctuating.

I could compare the minimum recording time with the minimum zsh-bench time for each framework, but let's use the averages since the stdev of zsh-bench for zim was so high. So for completeness of information, the first_prompt_lag_ms average for zsh4humans is 30.213ms, with a stdev of 0.344. And in recordings the average was 244.444ms (stdev was 50.918, but again it was actually just one frame more in one of the recordings).

Then, it's odd that zim is 1.703x faster in zsh-bench than the recordings -- I was expecting much closer values --, and zsh4humans is 8.091x faster in zsh-bench than the recordings!

If anything, zsh-bench if favoring zsh4humans by a lot, both in terms of more stable measurements, than giving it times way faster than the real ones.

[romkatv]

If anything, zsh-bench if favoring zsh4humans by a lot

To my knowledge, it does not.

The procedure you've used for benchmarking is underspecified, so I cannot replicate your results. It also doesn't model real-life use of zsh. Starting login shell and immediately closing it is quite unusual, and so is overriding HOME. It's cool that you want to validate zsh-bench results, and I'd love to see that happening, so let's figure out a way to do it right.

It's vital to perform all measurements in a container to allow everyone to create the same environment. I've created a script that spins up a docker container and sets up zsh in it. You can use it like so:

curl -fsSLO https://raw.githubusercontent.com/romkatv/zsh-bench/master/playground
chmod +x ./playground
./playground zim

Instead of zim you can pass zsh4humans or any other name of a predefined config.

After a few minutes you should have interactive zsh with ~/repo as the current working directory. This is a git repository with 1k directories and 10k files.

You can install additional software with sudo apt install <pkg>.

If you want to record a video of starting a login zsh, use this command:

clear; print start; zsh -li

Once "start" appears on the screen, start the timer. Once "repo" (the current directory) appears, stop it. That's first prompt lag.

If you decide to base your benchmark results on a video recording, please specify the exact command that everyone can use to record video the same way. Any non-reproducible step in your methodology will void the results, so be precise.

Instead of video you can record raw output of zsh to the TTY complete with timing of all writes. This is what zsh-bench does. Here's how you can start a login shell and capture its writes to the TTY:

clear; script -fqet$HOME/timing ~/typescript

Run a few commands and exit zsh with exit as usual. Afterwards you can look at the raw output to the TTY:

cat -v ~/typescript | less

And the timing data:

less ~/timing

The format of the timing data is documented in man script:

The log contains two fields, separated by a space. The first field indicates how much time elapsed since the previous output. The second field indicates how many characters were output this time.

You can replay a recorded session like this:

clear; scriptreplay -t ~/timing ~/typescript

Don't resize your TTY or it won't replay properly.

I would very much prefer if you share ~/typescript and ~/timing instead of a video. They are much easier to work with.

You can also try using asciinema instead of script to record and replay a session. I don't know what kind of latency it adds on its own but in terms of capabilities it should work just as well as script. The advantage of asciinema is its output format -- a single json file with data and timing. In addition, asciinema sessions can be easily uploaded and watched in a browser.

You can run zsh-bench in the container like this:

~/zsh-bench/zsh-bench --iters 16

All benchmark data that I've published was acquired with --iters 64 but it takes a long time to run. With --iters 16 results are also decent.

You can edit rc files if you need to test something (install nano or vim for this). Restart zsh with exec zsh afterwards. ~/zsh-bench/zsh-bench will be affected by these changes, too.

If you find anything unexpected, or if your measurements match that of zsh-bench, please share either way.

Also from zsh-bench's code looks like this output is the min value.

Correct.

It would be good to know what is the stdev of values inside a zsh-bench run...

If you send a clean PR, I'll merge it.

... to make sure the values are not fluctuating.

If you take a minimum of several runs, these fluctuations won't matter.

let's use the averages

Minimum is better in benchmarks of this kind, especially when ran on commodity hardware with CPU scaling, background services and whatnot. Minimum fails only when workloads are different on different iterations but in these benchmarks they are the same.

[romkatv]

Here are sample results from my dev machine. I've abbreviated all prompts to % .

Machine info:

% grep 'model name' /proc/cpuinfo | head -1
model name      : AMD Ryzen Threadripper 3970X 32-Core Processor

% uname -sm
Linux x86_64

Setup:

% curl -fsSLO https://raw.githubusercontent.com/romkatv/zsh-bench/master/playground
% chmod +x ./playground

Zim:

% ./playground zim
==> starting ubuntu:impish in docker ...
==> setting up a container ...

% ~/zsh-bench/zsh-bench --iters 64
creates_tty=0
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=116.601
first_command_lag_ms=133.514
command_lag_ms=96.829
input_lag_ms=11.980
exit_time_ms=28.159

% clear; script -fqet$HOME/timing ~/typescript

% exit  # exit from `script`

zsh4humans:

% ./playground zsh4humans
==> starting ubuntu:impish in docker ...
==> setting up a container ...

% ~/zsh-bench/zsh-bench --iters 64
creates_tty=1
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=14.943
first_command_lag_ms=55.096
command_lag_ms=2.366
input_lag_ms=4.725
exit_time_ms=5.969

% clear; script -fqet$HOME/timing ~/typescript

% exit  # exit from `script`

Script logs:

script-logs.tar.gz

Note that zsh4humans starts tmux, as indicated by creates_tty in the output of zsh-bench. You can disable this with the following command:

sed -i -E '/start-tmux/ s/integrated/no/' ~/.zshrc

[ericbn]

The procedure you've used for benchmarking is underspecified, so I cannot replicate your results.

The only thing I haven't specified is what I've used to do the screen recording. I've used Kap, but any screen recording software will do.

It also doesn't model real-life use of zsh. Starting login shell and immediately closing it is quite unusual, and so is overriding HOME.

Starting a login shell and immediately closing it is enough to measure the time for the first prompt to appear. The prompt should not take a different time to appear because the first command after it is exit. I'm starting zsh with HOME=/path/to/new/home zsh -il to have a shell with a different home, which can be unusual because we usually don't need a different home, but is perfectly valid. You'll get a perfect "real-life" zsh shell with a new home with that, which is what is needed to benchmark the different frameworks.

It's cool that you want to validate zsh-bench results, and I'd love to see that happening, so let's figure out a way to do it right.

It does not look like you'd love to see that happening, as you didn't even bother to understand how to replicate my results, and already tried to minimize what I did in every sentence of your paragraph above.

Once "start" appears on the screen, start the timer. Once "repo" (the current directory) appears, stop it. That's first prompt lag.

That's exactly what I did. I just printed the framework name instead of "start", which I hope you agree should not make any difference.

Minimum is better in benchmarks of this kind, especially when ran on commodity hardware with CPU scaling, background services and whatnot.

This is just because you say so. You find it very easy to say anything just to keep your current position.

EDIT: Actually it's recommended to compare all data points instead of just the average. See Don't Compare Averages. And before you say the article title means that it's better to use minimum instead of averages, it does not. The article starts with "In business meetings, it's common to compare groups of numbers by comparing their averages", and then goes on from there.

[ericbn]

Note that zsh4humans starts tmux, as indicated by creates_tty in the output of zsh-bench.

You can start whatever you want in you framework, and whatever runs as part of the shell startup until the first prompt appears should be counted as the time from the shell start to the first prompt appearance.

[romkatv]

The only thing I haven't specified is what I've used to do the screen recording. I've used Kap, but any screen recording software will do.

In order to replicate your results I need the full list of commands that I can run on a freshly installed OS.

Starting a login shell and immediately closing it is enough to measure the time for the first prompt to appear. The prompt should not take a different time to appear because the first command after it is exit.

If a benchmark doesn't measure real life performance, it's a problem with the benchmark. You've added sleep 1 to zim benchmark for the same reason. Starting zim in a tight loop has terrible performance but it doesn't matter because in real life no one starts login shell in a tight loop. You don't need to "fix" zim in this regard -- it's not broken. So fixing the benchmark was the right call. We want to measure real life performance, so we adapt the benchmark until it starts measuring real life performance. We don't change real life to adapt to a benchmark.

I'm starting zsh with HOME=/path/to/new/home zsh -il to have a shell with a different home, which can be unusual because we usually don't need a different home, but is perfectly valid. You'll get a perfect "real-life" zsh shell with a new home with that, which is what is needed to benchmark the different frameworks.

The closer you get to real life usage, the less likely that you are measuring wrong things. In real life people don't override HOME, so it's better to not do that. If you can demonstrate that overriding HOME never affects benchmark results, then you can override HOME but it makes the benchmark more brittle. If my benchmark doesn't override HOME and yours does, and our results disagree, my results are more authoritative because my benchmark better models real life usage. Let's stay as close as possible to real life and let's specify benchmark conditions as fully and precisely as possible. It's much easier to argue about code when we both can run it than it is to argue about words. Easier to argue implies easier to sort out differences and come to an understanding of where the discrepancies in the measurements lie.

It does not look like you'd love to see that happening, as you didn't even bother to understand how to replicate my results, and already tried to minimize what I did in every sentence of your paragraph above.

For the sake of efficiency, let's stay on the subject matter and avoid speculations about each other's motivation, character, attitude, etc.

I'm willing to replicate your results if you give me instructions as specific as the ones I gave you. It should be literally "start with a freshly installed OS and run this series of commands". All commands must have their source available. I would greatly appreciate if the OS is also free and open source -- Linux, FreeBSD, etc. If I get different results when I run the same commands as you, we'll narrow it down to things we've failed to control (machine, cosmic rays, etc.).

Another way to do this is to start with the specific commands that we already have -- the ones I posted in my previous comment. I can run them and you can run them. You don't have to accept the numbers printed by zsh-bench as authoritative. Go ahead and record TTY on video, or via script, or via asciinema. If you find a mismatch in numbers by using a different methodology, post the instructions for me to replicate your measurements (ideally packaged as a script). It would save a lot of time if you start with the container I've created but if you want to create your own, that's also fine. Just post the code so that I can have the same container on my machine.

EDIT: Actually it's recommended to compare all data points instead of just the average. See Don't Compare Averages. And before you say the article title means that it's better to use minimum instead of averages, it does not. The article starts with "In business meetings, it's common to compare groups of numbers by comparing their averages", and then goes on from there.

Let's postpone this discussion. It's significantly less important but it can derail us.

You can start whatever you want in you framework, and whatever runs as part of the shell startup until the first prompt appears should be counted as the time from the shell start to the first prompt appearance.

This is obviously true. I'm probably missing a point you are trying to make here.

[romkatv]

After reading the issue description a few more times here's my theory why you see the discrepancy in first prompt lag as reported by zsh-bench and as measured by yourself.

Firstly, when you run ./zsh-bench zim, it runs in docker, hence on Linux kernel. Performance on Linux can be very different from performance on macOS (I'm assuming your host OS is macOS). In addition, zsh startup files often take different code path on different operating system. They are also affected by the availability of external commands installed on the machine. I think if you run this:

curl -fsSLO https://raw.githubusercontent.com/romkatv/zsh-bench/master/playground
chmod +x ./playground
./playground zim

... and then run clear; print start; zsh -li while recording your screen, you'll see agreement with the numbers from ~/zsh-bench/zsh-bench --iters 16 (you need to run the latter command also in the container). Those numbers will also be lower than what you seen on your macOS host. I highly recommend avoiding buffering input before starting shell. It doesn't improve the quality of the benchmark at all and only introduces complexity. Type exit after you see prompt.

There is also the issue with initialization of zsh in the procedure you've used. I already mentioned it in zimfw/zsh-framework-benchmark#8 (comment) (the second bullet point). In order to let zsh4humans initialize, you need to start it at least once and let zle enter idle state. In practice this happens naturally because people aren't sending buffered input with exit in it to interactive shell so that zsh terminates before it has a chance to read all of the input.

The most reliable way to warmup zsh after installing a config is by the following sequence:

% HOME=HOME=${PWD:h}/zsh4humans -li
% :
% exit  # run this 1s after the previous command completes

zsh-bench does this automatically. After this you can run your benchmark as before although I once again should mention that buffering "exit" as input isn't achieving anything useful.

Edit: When you are running zsh-bench with positional arguments -- as in zsh-bench zim -- there is no need to run it several times because this command already performs multiple iterations (you can specify how many with --iters N). You can also specify multiple configs at once: zsh-bench zim zsh4humans. It'll be faster because zsh-bench will set up the container only once. It'll still provide great isolation because zsh-bench deletes the test users and wipes /tmp before each benchmark. playground doesn't currently allow you to run both zim and zsh4humans in the same container, so you'll need to create a separate container for each.

[ericbn]

In order to replicate your results I need the full list of commands that I can run on a freshly installed OS.

Cool, let me work on that. I purposely want to keep the steps where the screen is recorded and the recording frames are verified as manual steps, as I want to make sure no automation is interfering with the results.

Performance on Linux can be very different from performance on macOS (I'm assuming your host OS is macOS).

Yes, I'm using macOS, and yes there seems to be more going on in the /etc startup files in macOS than in Ubuntu, which might explain why zsh-bench reports a quicker time than the recording I did in my macOS. What do you think about running the benchmarks with zsh -o NO_GLOBAL_RCS -li in the future?

while recording your screen, you'll see agreement with the numbers from ~/zsh-bench/zsh-bench --iters 16 (you need to run the latter command also in the container).

Sure. Let me use your playground container, or another container as close to it as possible so the recordings can be equivalent to the expect results from zsh-bench.

In order to let zsh4humans initialize, you need to start it at least once and let zle enter idle state.

Good to know. I'll make sure to do that as the first step before actually doing the frames counting in the recordings.

there is no need to run [zsh-bench] several times because it already performs multiple iterations

I know. I just wanted to show how the results fluctuate between subsequent zsh-bench zim runs. And in comparison, my subsequent recording times didn't fluctuate as much.

I highly recommend avoiding buffering input before starting shell. It doesn't improve the quality of the benchmark at all and only introduces complexity. Type exit after you see prompt.

I thought you were measuring the first command lag like this: Type the command before the prompt appears, and measure how long it takes for the command output to actually appear after the prompt appears. (If we type it after the prompt appears, there's already a lag between the time the prompt appeared and the time the command was typed, at least when typing manually.) If not, then how are you measuring it? And which command are you specifically using in zsh-bench?

[romkatv]

I purposely want to keep the steps where the screen is recorded and the recording frames are verified as manual steps, as I want to make sure no automation is interfering with the results.

I'd rather have both of these automated and then we can also manually perform the same steps to double-check that the automation produces the same results.

(Frankly, I don't think recording a video is a wise approach. Recording the raw writes to the TTY is both easier and more precise. But if you want to go with video, that's fine with me. It's more work but the numbers should match eventually after we iron our potential quirks with vsync, adaptive frame rate, codecs, buffering, etc.)

Performance on Linux can be very different from performance on macOS (I'm assuming your host OS is macOS).

Yes, I'm using macOS, and yes there seems to be more going on in the /etc startup files in macOS than in Ubuntu, which might explain why zsh-bench reports a quicker time than the recording I did in my macOS.

You can also run zsh-bench without positional arguments to benchmark your own local login shell. You'll have to install script from util-linux though and I don't know how easy or hard that is on macOS. If you manage to instal it, create a local user, set up any kind of zsh environment you like and run zsh-bench to see how that shell performs on macOS. You can also use your regular user if you have a way to restore your rc files (e.g., from git).

What do you think about running the benchmarks with zsh -o NO_GLOBAL_RCS -li in the future?

The config called "zim" is for the default experience. It measures the performance of zim that a potential user would see if they installed zim and made no modifications. Similarly with "zsh4humans". The config called zim+ disables everything -- including global rcs -- and enables only the core functionality (what's considered core is in the docs of zsh-bench) with all possible optimizations. It shows the baseline latency for a config that one could build on top of zim. If zsh-bench ran all configs with no_gobal_rcs, it wouldn't be able to benchmark some configs that people actually use. E.g., it wouldn't be able to tell us what kind of performance people should expect if they installed zim or zsh4humans and used it without modifications.

I know. I just wanted to show how the results fluctuate between subsequent zsh-bench zim runs.

It looks like the first run was slow and the second two runs had identical first prompt lag. Perhaps something was running on your machine at the same time? Try using --iters 64, close all applications and don't do anything while the benchmark is running. As I mentioned, this is the value I used for all benchmarks for all published results.

I thought you were measuring the first command lag like this: Type the command before the prompt appears, and measure how long it takes for the command output to actually appear after the prompt appears. (If we type it after the prompt appears, there's already a lag between the time the prompt appeared and the time the command was typed, at least when typing manually.) If not, then how are you measuring it? And which command are you specifically using in zsh-bench?

Yes, that's how zsh-bench measures first command lag. But you aren't measuring first command lag, you are measuring first prompt lag. If you want to measure first command lag, you need to buffer print first""-""command and then track when "first-command" appears on the screen. Note that first command lag depends on the length of the command if syntax highlighting is enabled. Buffering "exit" doesn't allow you to measure first command lag and doesn't help with measuring first prompt lag.

I forgot to mention that you can replay the screen of the TTY that zsh-bench is acting on. Let's say you are in playground zim. Run this:

rm -rf -- '' /tmp/zsh-bench-*(D) && ~/zsh-bench/zsh-bench --iters 1 --keep

Edit: I've added the crucial --keep to the command above.

Once this command finishes, you can see what zsh-bench was doing:

clear; scriptreplay -t /tmp/zsh-bench-*/timing /tmp/zsh-bench-*/out

This works only if your TERM is compatible with xterm-256color. You can multiply and/or increase by a constant all numbers in the first column of /tmp/zsh-bench-*/timing for slower replay.

This method is very useful when debugging zsh configs. It allows you to see errors if there are any and to generally verify that things work as they are supposed to.

You'll see that at first zsh-bench issues a short command that sources something. That's the first command and it gets sent to the TTY right away without waiting for the first prompt. The sourced file prints, and the result of that printing signifies execution of the first command.

Then zsh-bench starts measuring input latency. It prints a fairly long command composed of a bunch of escaped abc tokens (they are escaped to avoid triggering alias expansion, which can significantly slow down syntax highlighting), types one more character and waits for that character to be processed by zle. This is performed several times and then the command line is cleared without execution.

Then zsh-bench measures command lag by spamming ENTER and seeing how many prompts it would get.

Once you play around with script and zsh-bench a bit and verify some numbers with video recording, it shouldn't be hard to see how zsh-bench measures things and that the way it measures it corresponds to what we actually want to measure. You can even run clear; print start; script -fqet$HOME/timing ~/typescript in the container and record a video. This will allow you to verify that latencies extracted from the logs of script match latencies that you get from video.

[Eriner]

Sorry, just catching up on all this and discussion in the other benchmarking repo.

@ericbn My memory is hazy and I don't remember if there is any interaction with login shells when using zpty, which my original script used, and if there is/was it may not be documented.

Awesome that you were able to generate so many stats. Reading all of this was really the best kind of issue report :)

@romkatv this is a really cool tool, although I'll admit I haven't looked deeply into your zsh4humans project. If the claim to fame is that it is fast, awesome! Fixing the horrendously slow prompt lag on slower devices (first generation raspberry pis, old old intel chips, etc) that came with other frameworks was one of my goals when I started zim.

Part of the original design of zim was also to allow the "heavier" tasks to run at the login shell, with the idea that it usually only happens once per interactive session. That is to say, when you sit down and login or connect, you're only going to do it once per session/terminal. This included things like zcompile operations, initialization of the ssh/gpg agents, etc. My perception was, "if the user is forced to wait, make them wait one time and only one time".

In that same line referenced above, you can see that an explicit and seemingly superfluous zsh process is launched with the ZDOTDIR of the to-be-benchmarked framework rc's. This explicitly allowed first-time initialization of each framework (explicitly not benchmarking the first prompt lag). This isn't to say that it doesn't matter, it was just an intentional compromise.

Design obviously wasn't perfect, but was a good compromise vs trying to make things asynchronous until the cows come home. By happenstance, working on my original zsh framework benchmarking tool is where I learned precisely how painful that is ;)

My only question, do you run login shells frequently? If you open a new terminal is it always a login shell? I ask only because it doesn't match my personal experience.

I'd be interested to see these benchmarks run on a variety of devices. There is a huge difference in speed based on IO devices. This is why the original benchmarks for zim ran on a variety of devices. Machines with an NVMe will perform better with a Celeron chip than a machine with an HDD and a Ryzen Threadripper in some circumstances. (Zim was knowingly designed with file IO in mind, hence the zcompile of user RC files that you disagree with). You reference your CPU here, but not the disk: https://github.com/romkatv/zsh-bench#full-benchmark-data

cheers! 🍻

[romkatv]

My only question, do you run login shells frequently?

Not particularly. Why do you ask?

If you open a new terminal is it always a login shell?

Yes. Note that this is the default in Apple Terminal and iTerm2. You can change it but then .zlogin and .zprofile will never get sourced, not even on boot.

I'd be interested to see these benchmarks run on a variety of devices.

Cool. If you run the benchmarks on other devices, please share the results.

You reference your CPU here, but not the disk: https://github.com/romkatv/zsh-bench#full-benchmark-data

It's NVMe M.2. I added this information to the doc.

[romkatv]

I'd be interested to see these benchmarks run on a variety of devices.

Cool. If you run the benchmarks on other devices, please share the results.

Out of curiosity I've benchmarked zim and zsh4humans on a Raspberry Pi 4 4G Model B with a micro SD card for storage.

==> benchmarking zim
creates_tty=0
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=315.364
first_command_lag_ms=387.934
command_lag_ms=229.512
input_lag_ms=69.796
exit_time_ms=65.726

==> benchmarking zsh4humans
creates_tty=1
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=61.945
first_command_lag_ms=233.866
command_lag_ms=13.231
input_lag_ms=56.386
exit_time_ms=19.132

Edit: See more accurate results below.

[Eriner]

Edit: the contents below were edited by @romkatv. I, @Eriner, did not author any of the text below.

Re: the rpi4 benchmarks, I'd bet the results are even more pronounced for an rpi1

What do you mean by "pronounced"? Which conclusions did you draw from the benchmark results I posted above?

I reran the benchmarks today with proper isolation and --iters 64 to match the conditions of other published benchmark results. This time I also included no-rcs and allowed my pi to cool down between benchmarks to avoid thermal throttling.

Raw:

==> benchmarking no-rcs ...
creates_tty=0
has_compsys=0
has_syntax_highlighting=0
has_autosuggestions=0
has_git_prompt=0
first_prompt_lag_ms=0.028
first_command_lag_ms=0.876
command_lag_ms=0.212
input_lag_ms=0.993
exit_time_ms=10.249

==> benchmarking zim ...
creates_tty=0
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=315.910
first_command_lag_ms=386.317
command_lag_ms=227.338
input_lag_ms=69.142
exit_time_ms=65.413

==> benchmarking zsh4humans ...
creates_tty=1
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=58.987
first_command_lag_ms=222.734
command_lag_ms=12.957
input_lag_ms=27.730
exit_time_ms=19.110

Normalized:

config	tmux	compsys	syntax highlight	auto suggest	git prompt	first prompt lag	first cmd lag	cmd lag	input lag
no-rcs	❌	❌	❌	❌	❌	0% 🟢	1% 🟢	2% 🟢	5% 🟢
zim	❌	✔️	✔️	✔️	✔️	632% 🔴	258% 🔴	2273% 🔴	346% 🔴
zsh4humans	✔️	✔️	✔️	✔️	✔️	118% 🟠	148% 🟠	130% 🟠	139% 🟠

For comparison, here are the results from my desktop:

config	tmux	compsys	syntax highlight	auto suggest	git prompt	first prompt lag	first cmd lag	cmd lag	input lag
no-rcs	❌	❌	❌	❌	❌	0% 🟢	0% 🟢	1% 🟢	1% 🟢
zim	❌	✔️	✔️	✔️	✔️	252% 🔴	95% 🟡	974% 🔴	61% 🟡
zsh4humans	✔️	✔️	✔️	✔️	✔️	20% 🟢	34% 🟢	24% 🟢	24% 🟢

On the pi all latencies are between 2.3 and 5.9 times higher. That's a fairly narrow range considering the differences between these machines on different dimensions: CPU architecture, single core performance, number of cores, I/O, etc. It's worth noting that the pi performs remarkably well.

[romkatv]

Cool, let me work on that.

Hey @ericbn, have you made any progress on that?