Skip to content

Conversation

@akx
Copy link
Contributor

@akx akx commented Jan 21, 2019

Following up on #367, I decided to try out whether it'd be feasible to add an SQLite backend for multiprocess values.

It's generally a little slower writing values, but a whole lot faster reading them out, especially when there are many PIDs! (Full disclosure, though: I'm only emulating multiple processes by hacking os.getpid() in my test bench, so these numbers could be off in a real-life situation.) See below for results.

⚠️ This hasn't yet been tested in a production situation at all, nor profiled – I'm sure there are some suboptimal bits and pieces in there. I felt like it would be better to post a work-in-progress PR anyway.
(Come to think of it, another avenue to investigate, if SQLite is considered too heavy-handed, would be some sort of append-only structure of files that can be easily collated and merged.)

prometheus_sqlite

akx added 2 commits January 21, 2019 18:48
Signed-off-by: Aarni Koskela <akx@iki.fi>
Signed-off-by: Aarni Koskela <akx@iki.fi>
akx added 4 commits January 22, 2019 11:38
Signed-off-by: Aarni Koskela <akx@iki.fi>
Signed-off-by: Aarni Koskela <akx@iki.fi>
Signed-off-by: Aarni Koskela <akx@iki.fi>
Signed-off-by: Aarni Koskela <akx@iki.fi>
@brian-brazil
Copy link
Contributor

if SQLite is considered too heavy-handed, would be some sort of append-only structure of files that can be easily collated and merged.

That'd be even slower. I'd like to keep to no dependencies for this, as that makes it easier to adopt the library. Any tricks that sqllite is pulling, we can potentially pull too.

I'm suspicious of these results, I'd expect a lot more overhead from SQLite and the GIL is probably avoiding contention here.

@brian-brazil
Copy link
Contributor

This has gone stale, and doesn't make sense to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants