Move the core of the combine logic to be entirely in SQL

[alex]

This shaves roughly 40% of the runtime off of pyca/cryptography's combine.

A major limitation is that it currently doesn't work with in-memory sqlite databases, I'm not sure if there's a good way to do that.

[alex]

Also a few tests are failing 😂.

Before I run too much farther on this path, wanted to see how you feel about it?

[nedbat]

It's an interesting idea, thanks for starting it. What in-memory databases are you concerned about?

[alex]

on_disk=True on CoverageData

[nedbat]

Right, i know the API makes it possible, I'm just wondering realistically when it happens.

[alex]

Oh, I have no idea. I just assumed I wasn't allowed to break it either way :-) I think there's a way I can make it work though.

[alex]

This should be ready for review now, all green :-)

[nedbat]

Looking at a few recent runs, this seems to make my combine step slightly slower, though these are hardly precise measurements (with this: 2m 57s, 2m 59s), (without this: 2m 35s, 2m 48s). How many files are you combining, and what are the times?

[alex]

29 .coverage files + 30 lcov files (that I pre-merge into a single CoverageData), in total its ~415MB of stuff.

On current coverage its consistently ~15 seconds and with this patch its consistently ~9 seconds. (Measured 5x times)

I'm guessing you must have a lot more data for it to nearly 3 minutes? I can profile (and maybe do some SQL EXPLAIN) on these queries to see where the offender is, though it may not be representative of your workload.

[alex]

With some quick instrumentation, I can see that ~85% of my time is being spent merge the arcs data, are you using branch coverage for yours or line?

[nedbat]

I am using branch coverage, but I was also quoting the time for my "combine and report" step. Splitting it into parts shows that combine is not the problem:

Time for load: 0.00s
Time for combine: 2.34s
Time for save: 0.00s
Time for load: 0.00s
Wrote HTML report to htmlcov/index.html
Time for html: 22.74s
Wrote JSON report to coverage.json
Time for json: 120.34s

I have to assume my timing difference is noise. (the coverage.json is 2.8Gb, and I don't even keep it from CI!)

[alex]

Well, investigating I found another ~2-2.5 seconds to save, I'm now down to 7s consistently.

[nedbat]

It's good: before this: 3.59s, 3.56s. with this: 1.47s, 1.51s. Excellent

[alex]

Perfect!

[nedbat]

This was amazing, thanks so much!

[alex]

Thank you!

[nedbat]

This is now released as part of coverage 7.10.5.

[alex]

Awesome, thank you! All that is necessary for evil to succeed is for good people to do nothing.

…

On Sat, Aug 23, 2025, 10:45 AM Ned Batchelder ***@***.***> wrote: *nedbat* left a comment (nedbat/coveragepy#2033) <#2033 (comment)> This is now released as part of coverage 7.10.5 <https://pypi.org/project/coverage/7.10.5>. — Reply to this email directly, view it on GitHub <#2033 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBDEOOJAQJL67RRX5H33PB47PAVCNFSM6AAAAACEDEXCTCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMJXGA3TEOBRGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>