Azure CI brainstorming

[ggerganov]

If we assume we had enough MS Azure credits, how difficult would it be to make a llama.cpp / whisper.cpp / ggml CI in the cloud that runs on every commit and does some performance and perplexity benchmarks across a variety of hardware and models?

I don't have any experience in this, so would like to hear some opinions and maybe also see if there are people that would be interested in implementing such type of CI.

[ianscrivener]

@ggerganov,
A good idea and pretty easy/doable! I'd be happy to do this and submit a PR.

Let's iterate the solution on llama.cpp, then copy/paste the solution to whisper.cpp & ggml when it is "good enough"

TBH: I've been wanting to contribute to llama.cpp... but haven't had the matching tech skills until now.

[AlphaAtlas]

If you can secure one, Oracle's free Ampere instances would be a good candidate for continuous testing on ARM. Its 4 Ampere cores, 24GB, and enough ssd storage space for a few models.

[AlphaAtlas]

Also, one issue with VM cloud benchmarking is that performance is inconsistent.

If you do twenty runs of the same benchmark, the performance spread could be huge. Then you might spin up the exact same type of instance the next day, run twenty more runs of a benchmark, and then performance is very consistent.... but somehow slower than the previous sporadic run. I am not sure why this is the case, but I'm sure it has to do with other tenants stressing the CPU and its subsystems, variable silicon binning and such.

So if performance metrics are important, consider a small bare metal instance?

The Oracle VMs previously mentioned suffer from this, but since they are free and 24-7, you can probably get around it with a huge number of tests.

[ianscrivener]

Good point.
Yes, bare metal / dedicated VM is essential for permormance testing

[ianscrivener]

I've got cloud accounts with Google, Amazon & Microsoft Azure... a lot of free capacity with Azure & Google

Curiously, I've had problems signing up with Oracle (in Australia). Tried twice unsuccessfully, not sure if it my email address, credit card other?

AWS also has ARM machines - Inferentia & Trainium. IMO with the nvidia shortages Amazon's custom silicon will be getting a lot more more popular. Huggingface apparently use them for most (all?) training.

@ggerganov mentioned Azure - so I'll do a POC there. Assume he has some freebie on offer. I just signed up for USD$4500/year with Azure for a non-profit I'm helping out.

AWS has bare metal Apple M2s - (later on) I'd like to do performance testing there... give than Apple Silicon is a "first class citizen" for llama.cpp. They have a minimum 24hours usage purchase - so there are costs involved unless we get a ggml.ai AWS freebie deal.

[ianscrivener]

MacStadium have baremetal Apple M2s for around USD$120/month - https://www.macstadium.com/pricing

Perhaps a quick/tactical solution for perpexity, latency & resources benchmarking?

[ggerganov]

I should get access to "Azure Credits", but still not sure what this means exactly. Will follow-up when I have more info

[howard0su]

Be aware that you will get different VM every time you create a new VM. In order to make the perf result stable, you may have to keep one VM running.

[ianscrivener]

Yep! We probably should go bare metal for performance testing... and machines that we benchmark with over months & years.

[mechanicmuthu]

I'm a nobody in these topics, but I believe

1. bare metal
2. keeping VM running
3. repeatable benchmarking
4. benchmarking using m2 dedicated servers, etc
   are all taking us away from the intent here. The intent is to run llama reliably in ANY cloud instance. (with the benchmarking per run for info). Users will choose whether they can afford these things themselves. Thats my view.

[ianscrivener]

Agree @mechanicmuth..

I'm still waiting for @ggerganov to organise dedicated test machines...

in the mean time I might code something slightly different... that does perplexity only

BTW: everybody is somebody!

[gjmulder]

👍 for bare metal, unless you're willing to run multiple extended tests and report percentile summaries.

Since llama.cpp is primarily bottlenecked by memory I/O, running on any shared virtualized environment means llama.cpp's performance can be randomly throttled by memory I/O from other coscheduled VMs. AFAIK most if not all virtualization solutions do not provide any memory I/O throughput guarantees, unlike virtualized CPU and network throughput.

[ianscrivener]

@gjmulder, agree Gary.
IMO perplexity latency benchmarking is pretty much useless unless we have quality bare metal or dedicated host.

@ggerganov said he can get infinite Azure... so I was thinking small dedicated host with small Nvidia GPU. Have researched this in depth.

Even whole machine GPU cloud services like RunPod, Vultr et al offer a Docker container on a mostly (but not garranteed dedicated) machine. If it was dong backups or updates if would effectively be a "noisey neighbour" situations and ruin the perpexity latency benchmarks credibility.

AWS has Mac Mini baremetal, but you need to buy a whole day's worth of time. Again, budget is required.

Recently MS Azure refused GPUs to a not-for-profit I'm helping out who has USD3.5k of MS credit. So I'll believe free bare metal or dedicated host from GPU from Azure when I've spun it up and sshed in and tested it.

I'm reluctant to start coding a perplexity score + perpexity latency banchmark systems without test hardware locked in. Code would be mostly useless without dedicated test hardware.

Though I have been pondering a different approach.. perplexity scores only, post scores to HuggingFace Data or somewhere so anyone can run benchmark perplexity and anyone can Python/Jupyter graphs of llama.cpp perplexity results. Perhaps even a distributed job queue to fill in the benchmark data gaps.

[ianscrivener]

@ggerganov,

OK, so I have had an initial review, checked out the "perplexity page", reviewed the Github actions to build the releases, and test run ./perplexity etc.

Some questions - please pls confirm/correct my assumptions

Q1: Test Variables

1. llama.cpp release id - eg master-bd34cdd
2. hardware - eg Mac Mini 2022 M2 16Gb
3. model - eg /models/7B/ggml-model-q4_0.bin
4. OS+GPU+Engine: MacOS-M2-Metal | Windows-CUDA-OpenBLAS | Linux-CUDA-BLAS .. etc etc
5. ?? context size - eg 512, 1024, 2048 (unsure - pls confirm)
6. ?? other command line options (unsure - pls confirm)

Q2: Test Results Required

1. perplexity score - eg 4.2701
2. perplexity test duration - eg 3.74 hours, or 13,464seconds.. according to the perplexity process
3. ?? time according to time (unsure - pls confirm)
4. ?? full ./perplexity process stdout (unsure - pls confirm)

Q3: How will we present/display the results

1. web page / web dashboard
2. Jupyter notebook
3. rendered static graphs to be inserted in markdown READMEs
4. all of the above (in time)

Q4: Where/how to record test results?

1. database
2. Github as json, csv, other
3. all of the above

Q5: Test Hardware & OS

1. MacOS M2
2. Intel + Nvidia GPU - Windows
3. Intel + Nvidia GPU - Ubuntu
4. AMD + AMD GPU - Windows
5. AMD + AMD GPU - Ubuntu
6. ARM Ampere (cloud) - Ubuntu
7. Raspberry Pi (edge) - Raspbian
8. Smaller Arm - eg Nuc (edge) - Ubuntu
9. ?? other

[SlyEcho]

Yes, I was suprised to see ETA 3 hours when I first ran ./perplexity

There has been some mention of using a truncated wiki.test.raw with only the first 406 lines.

[ianscrivener]

thanks @SlyEcho .. I'll post a question on that Discussion page

[ggerganov]

Q1: Will we want to run multiple bit size/quantiztions of the model: FP14, q4_0 etc ?

Yes - ideally, run all quantization types. Could try to design the CI to have "frequent" runs of a single quantization type and "rare" (i.e. daily / nightly) runs on the rest to save time.

Q2: would we run (in time) different sizes of the models: 3B, 7B, 13B?

Yes - probably even bigger models if we can apply the same "frequent" / "rare" strategy from above

[ianscrivener]

👍🏻 OK, I like the tiered approach to CI priorities

[ggerganov]

a completely separate project & git repo

I made a CI team and repo in ggml-org - can invite anyone interested in collaborating on this project.
Let me know and I'll send an invite:

• repo: https://github.com/ggml-org/ci
• team: https://github.com/orgs/ggml-org/teams/ci

Alternatively - I'm also open to other suggestions for hosting / developing the framework

[gjmulder]

@ianscrivener, replying here as this about performance and not perplexity per se:

my understanding (perhaps wrong) is that we are aiming to get
(1) perplexity score - eg 8.2964 (per my test above)
(2) perplexity test runtime - eg 360.09s (per my test above)

for a given a combination of
(a) model - eg open-llama-3b-q4_0.bin
(b) hardware +GPU Library - eg Mac M2 2022 - 8 CPU cores (4 performance, 4 efficiency) - 16Gb RAM + Metal
(c) test data - eg wiki.test.raw.406
(d) context size - eg 512, 1024, 2048

If you're looking for performance (2) your dependent variable is execution time. (a), (c), (d) should be kept constant and you should vary (b). (1) is irrelevant then, as for CI/CD it shouldn't change (at least say to n decimal places - e.g. you might find minor differences in math libraries across different (b)). You could however simply test that the final perplexity doesn't change as a simple functional regression verification.

My point was that you may want to judiciously choose (c) and your llama config params so you get timings for all of the 1 = sample, 2 = prompt and 3 = eval time and 1 + 2 + 3 = total compute time. Ignore the load time as that's a function on the model size and disk I/O of the VM the model is running on. Perplexity looks to only test load and prompt time. @ggerganov can confirm if just prompt time is sufficient for a performance test.

Furthermore, you want to ensure the runs are of sufficient duration that the potentially high level of variability between different successive virtualized instances will be somewhat averaged out. I'd use just 2 (or maybe 4) vCPUs (i.e. -t 2) to attempt to reduce variability between successive runs, as the more vCPUs you ask for the more chance of contention between your performance test VM and other co-hosted VMs. I'd look to configure the runtime of each test for at least 10 minutes, but around 60 minutes would be better if you have the compute. One way to determine how much test duration you can/should use is look at the approx. frequency of commited PRs (e.g. one per 6 hours) and ensure your performance test completes before the next PR is committed. I've seen cases where multiple rapid PR commits DoSed the CI/CD compute infrastructure. I'm not sure if Github Actions queues test runs if the previous run is still executing.

Then, to deal with the inevitable variability of running on VMs when looking for non-functional performance regressions, I usually calculate an exponentially smoothed time series model of previous CI/CD total performance test run time results for each (b). You can then compare the last build for each (b) to this average and fail the build on performance if it is say 0.25 s.d. outside the predicted performance of the (b). I check both for upper and lower bounds as I've seen performance regressions where performance tests take 0 seconds to run 🤦

[ianscrivener]

A good idea...
I have considered this approach... though we'd be dependent on the GPU support that Docker provides.😞

I love Docker and it is one of my most used techs (spent all day doing Docker today). But...

Getting Docker to handle 6-9 month old nvidia GPUs is fine... but I reckon newer nvidia GPUs and non-nvidia GPUs would be a major PITA. And forget about AWS Inferentia and other ARM silicon. MacOS inside Docker with Metal GPU working well - I doubt we'll ever see that become a reality. Remeber that MacOS is a "first class citizen" for llama.cpp

"Inference at the edge" - yes Raspberry Pi has been mentioned for test hardware,

IMO baremetal is best!

[gjmulder]

Baremetal solves a lot functional and non-functional problems - if you have the budget to obtain it.

I spent most of my performance testing professional life constantly needing to control for variables such as virtualized compute, variable network performance, etc. as I worked in large corporate environments where if you wanted physical tin you had to convince your manager to buy it and then hide it under one's desk 🤣

[ianscrivener]

Agree. Baremetal is best for performance comparisons.
@ggerganov 's spec said "enough" of Azure credit.
Most baremetal GPU can be purchased by the minute at less than USD$1/hour.

[ggerganov]

I like the discussion!

Perplexity looks to only test load and prompt time. @ggerganov can confirm if just prompt time is sufficient for a performance test.

We can start with just prompt time of perplexity tool. At some point we might want to look into eval time of main tool as well, but to avoid complicating things at the start, lets focus just on perplexity for now.

@ggerganov 's spec said "enough" of Azure credit.

Yes - for now lets operate on the assumption that we will have "infinite" amount of Azure credits.

[ianscrivener]

OK.. "infinite" would probably be enough 👍 😀

[ianscrivener]

Some back of the ballpark guestimates & assumptions

a) Models        3	  Open Llama 3B, 7B, 13B
b) Context size  3	  512, 1024, 2048
c) Test data     1	  wiki.test.raw.406
d) Hardware      12       MacOS Metal, MacOS OpenBLAS, Nvidiax5, AMDx3, Inferentia, another ARM
                          = 108 tests
                          @ 5 mins per test (guess #1)
                          = 540 mins testing per release
                          = 9 hours testing per release 
                          @ $0.50/hour average (guess #2)
                          = $4.50 per release

[gjmulder]

5 minutes average per run looks like a good guesstimate for your CuBLAS and similar accelerated runs. I'd factor 10x that time for your OpenBLAS runs.

Here's a Gsheets matrix of perplexity runs @TheBloke and I performed with a selection of FB llama-based models. The "perf" column (in minutes) was the estimated perplexity runtime as reported by ./perplexity. The efficiency = 1 / (perf * perplexity) column was a metric I devised to compare the quality versus throughput trade-off for each result. There's numbers there for early BLAS enabled non-GPU accelerated runs at the bottom.

[ianscrivener]

brilliant! the spreadsheet help with my learnind curve. Thanks

[ianscrivener]

@gjmulder - question: what is the "perf" column on your spreadhseet please?

[gjmulder]

@ianscrivener the "perf" column (in minutes) was the estimated perplexity runtime as reported by ./perplexity

[ianscrivener]

@gjmulder - copy that. Thanks

[ianscrivener]

@ggerganov - I just dropped you an email regarding CI 😀

[tpaviot]

Hi guys, I see this issue related to Azure and CI. FYI, I set up a build/test/deploy workflow on my free azure account, originally to be able to build/download quantized models directly on azure and save time/storage/memory on my laptop. It could also serve as a basis for CI.

https://github.com/tpaviot/llama.cpp-azure is a separate project so far, I just changed it to public a couple of hours ago (it used to be a private repo), it could be a dev branch as well.

[ianscrivener]

thanks for sharing @tpaviot 👍

[ggerganov]

Thanks and sorry for delay on this topic - will try to setup things soon. I have the access now to the Azure Credits - need to learn how to create the VMs and use them

[ianscrivener]

@ggerganov,

I can setup things up if you give me Azure access... I'll email you again

BTW: usually you need to request access to GPU servers - they have safety limits set so people don't spends thousands accidently
It takes a day of so to get activated...

NCasT4_v3-series are a good shared GPU VM using T4.. to start playing around with

For proper latency benchmarking we need an Azure 'dedicated host' - ie a whole server machine - no noisey neighbours. Likely you should talk to your MS rep about this.. it will be a special order on a freebie plan. Beware; it seems to be standard MS Redmond policy for "no GPUs on freebie plans"... but likely they'll carve out an exceptino for llama.cpp

(Non GPU) Dedicated Host info is here - https://azure.microsoft.com/en-au/pricing/details/virtual-machines/dedicated-host/.. from about USD$2800 per month

[ggerganov]

@ianscrivener and all

Thanks for the info and apologise for not responding to email

I went ahead and implemented a custom CI framework with a few Bash scripts:

• https://github.com/ggml-org/ci
• ci : integrate with ggml-org/ci ggml#393

For now I plan to spin-up a few VMs (x86 and Arm) and polish the CI in the ggml repository - add more tests, coverage, text generation. After I feel it is in a good state, will expand it to llama.cpp and whisper.cpp and look into renting GPU and bare-metal instances for performance and perplexity tests.

This framework is a bit of a hack, but it allows me to have control and good understanding of how it works. This way I can iterate more quickly on it.

[ianscrivener]

@ggerganov,
yes... sounds like the best idea. CI/CD needs to be tightly coupled with development and probably setup by the developers. Azure CI sounds good... but is very bloated. Good for big corporates - but not so much for a fast paced opensource project with a distrubuted team. Github Action, Bash and Docker are more suitable tools.

You already have perplexity bash scripts... so can get scores per release... but it is manual to record that. IMO what is missing is;

1. easily/automatically gathering the perplexity test results (removing the manual processing)
2. storing it centrally
3. presenting this data

That is the bit I have been playing with:
(a) python code that wraps ./perplexty and generates a json file of the perplexity test run, sending the json to a web API
(b) the API that takes the JSON and puts it in a data store (GoogleSheets for now)
(c) easy viewing of scores (Google Sheets for now - per @gjmulder & @TheBloke)
... I'll send a PR for when it is ready

Latency benchmarking is a whole new beast - and needs dedicated hardware to be empirical and to be meaningful over time.

👀 I love the architectural approach of lzbench.. put competing binaries into a wrapper processes and test them on the same hardware at the same time... so my fork of this idea is a "LLM inference engine head-to-head test engine"

[ianscrivener]

@tpaviot I checkout out your code... looks like a good start.

Here's the bash code I use to setup and run a llama.cpp perplexity test on Docker container with CUDA support: https://gist.github.com/ianscrivener/71bde7a2bfc92e8d217900229d78df51

[ianscrivener]

In response to @ggerganov's call for perplexity and latency testing for llama.cpp I've coded llama.cpp perplexity scorecard... a helper project to run and gather ./perplexity results, uploads JSON data to AWS S3 for review, analysis and charting.