Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Speed up create_hogql_database #27451

Merged
merged 4 commits into from
Jan 15, 2025
Merged

perf: Speed up create_hogql_database #27451

merged 4 commits into from
Jan 15, 2025

Conversation

timgl
Copy link
Collaborator

@timgl timgl commented Jan 11, 2025

Problem

create_hogql_database is slow (somewhere between 1-1.5 seconds per run for team 2), and we run it multiple times for a single query.

This should make every query we run 1-2 seconds faster.

Changes

  • add caching to create_initial_channel_type (on average this takes 0.3 seconds)
  • make sure we only run create_hogql_database once

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

How did you test this code?

Copy link
Collaborator

@mariusandra mariusandra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, though seems to have broken some tests. I assume the issues is that some tests flipping between PoE modes and other modifiers didn't discard self.database somewhere.

Copy link
Member

@robbie-c robbie-c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General principle sounds good!

To "yes and" this - I'd guess that most of the 0.3 seconds in create_initial_channel_type comes from parsing HogQL, if we had a good pattern for caching the results of parsing HogQL in general (before team-specific context or placeholder values were applied) we could probably find similar speedups in a bunch of trends and web analytics queries

@timgl timgl enabled auto-merge (squash) January 15, 2025 00:17
@timgl timgl merged commit 97af0ac into master Jan 15, 2025
92 checks passed
@timgl timgl deleted the speed-up-hogql branch January 15, 2025 00:33
Copy link

sentry-io bot commented Jan 15, 2025

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ AlertCheckException: unhashable type: 'list' posthog.tasks.alerts.checks.check_alert_task View Issue
  • ‼️ TypeError: unhashable type: 'list' /api/environments/{parent_lookup_team_id}/query/ View Issue
  • ‼️ TypeError: unhashable type: 'list' posthog.tasks.tasks.process_query_task View Issue
  • ‼️ TypeError: unhashable type: 'list' posthog.caching.warming.warm_insight_cache_task View Issue
  • ‼️ TypeError: unhashable type: 'list' posthog.caching.warming.warm_insight_cache_task View Issue

Did you find this useful? React with a 👍 or 👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants