Skip to content

Commit

Permalink
Add Buffer Cache documentation (#294)
Browse files Browse the repository at this point in the history
  • Loading branch information
seanlinsley authored Dec 3, 2024
1 parent dedaf6d commit f79d0d9
Show file tree
Hide file tree
Showing 7 changed files with 64 additions and 0 deletions.
4 changes: 4 additions & 0 deletions directory.json
Original file line number Diff line number Diff line change
Expand Up @@ -992,6 +992,10 @@
"title": "Tracking Postgres schema statistics in pganalyze",
"path": "/docs/schema-statistics"
},
"schema-statistics/buffer-cache": {
"title": "Tracking Postgres buffer cache usage in pganalyze",
"path": "/docs/schema-statistics/buffer-cache"
},
"schema-statistics/columns-hot-updates": {
"title": "Column statistics and HOT updates",
"path": "/docs/schema-statistics/columns-hot-updates"
Expand Down
Binary file added schema-statistics/buffer-cache-index.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added schema-statistics/buffer-cache-system.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
58 changes: 58 additions & 0 deletions schema-statistics/buffer-cache.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
---
title: 'Tracking Postgres buffer cache usage in pganalyze'
backlink_href: /docs/schema-statistics
backlink_title: 'Tracking Postgres schema statistics in pganalyze'
---

Postgres caches recently-used table and index data in its buffer cache, whose size is configured by the `shared_buffers` setting. pganalyze can track buffer cache usage with the [pg_buffercache](https://www.postgresql.org/docs/current/pgbuffercache.html) extension, helping you to understand database access patterns over time and identify tables that may need to be optimized.

## Setup

Using collector version [0.63.0 or later](/docs/collector/upgrading), enable the extension:

```sql
CREATE EXTENSION IF NOT EXISTS pg_buffercache WITH SCHEMA public;
```

If you don't see stats after 20 minutes, check the collector logs for error messages.

If the collector runs into permissions issues when querying pg_buffercache, you may need to upgrade the extension:

```sql
ALTER EXTENSION pg_buffercache UPDATE;
```

Or on older Postgres versions, you can grant the permissions manually:

```sql
GRANT USAGE ON SCHEMA public TO pganalyze;
GRANT SELECT ON pg_buffercache TO pganalyze;
GRANT EXECUTE ON FUNCTION pg_buffercache_pages TO pganalyze;
```

## Performance implications of collecting shared buffer statistics

Since Postgres 10, the pg_buffercache view uses reduced locking and as such is generally safe to query on production systems without impacting the regular workload. pganalyze queries the buffer cache data every 10 minutes.

By default this feature is only turned on for servers with `shared_buffers` configured for 200 GB of RAM or less. You can change the threshold by setting the `max_buffer_cache_monitoring_gb` setting, e.g. to `300` in the collector config.

This threshold exists because pg_buffercache incurs a fixed CPU cost for each buffer page (more buffer pages = more CPU used by querying it), as well as the fact that it temporarily writes its data to a temporary file, which has a size of roughly 0.5% of the `shared_buffers` value, or about 1GB at 200 GB `shared_buffers`.

## Table Statistics

When viewing a specific table, you can see a breakdown of buffer cache usage for the table, TOAST, and index data over time. Here you can see an example of one table that's always in memory, while the other table is only in memory while its query workload requires it.

![](buffer-cache-table-constant.png)
![](buffer-cache-table-variable.png)

## Index Statistics

You can similarly see the buffer cache usage of a specific index:

![](buffer-cache-index.png)

## System Memory

The System Memory page has two graphs showing buffer cache usage across your Postgres server. Top 10 Tables shows the tables that usually take the most memory, while Top 10 Outliers shows the tables that use the most memory for a shorter period of time.

![](buffer-cache-system.png)
2 changes: 2 additions & 0 deletions schema-statistics/toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,5 @@
href: schema-statistics
- name: "Column statistics / HOT updates"
href: schema-statistics/columns-hot-updates
- name: Postgres buffer cache
href: schema-statistics/buffer-cache

0 comments on commit f79d0d9

Please sign in to comment.