Skip to content

Commit

Permalink
chore(changelog): reduce mem footprint
Browse files Browse the repository at this point in the history
The current Changelog structure consumes a significant amount of memory
due to the allocation of metadata for each field. As the number of
fields increases, the memory usage grows linearly. Approximately 240
bytes per field were observed just for metadata, excluding the actual
data and pointers for each field.

To reduce memory consumption, the new changelog.Entries implementation
uses a flat slice of any type instead of storing metadata for each field
separately.

---

| Caches | GOGC | Branch | Heap Use | Heap Growth | Delta   | Proctree |
|        |      |        | MB (Avg) |             |         |          |
|--------|------|------- |----------|-------------|---------|----------|
| -      | -    | main   | 28       | -           | -       | off      |
| 16384  | -    | main   | 199      | 181.00      | -       | on       |
| 32768  | -    | main   | 331      | 313.00      | -       | on       |
| -      | 5    | main   | 18       | -           | -       | off      |
| 16384  | 5    | main   | 125      | 107.00      | -       | on       |
| 32768  | 5    | main   | 209      | 191.00      | -       | on       |
| -      | -    | new    | 28       | -           | -       | off      |
| 16384  | -    | new    | 119      | 91.00       | -50.28% | on       |
| 32768  | -    | new    | 172      | 144.00      | -46.01% | on       |
| -      | 5    | new    | 18       | -           | -       | off      |
| 16384  | 5    | new    | 76       | 58.00       | -54.21% | on       |
| 32768  | 5    | new    | 111      | 93.00       | -48.69% | on       |

With GOGC set to 5, the new implementation reduces average heap usage by
approximately 54% when using cache sizes of 16,384. For cache sizes of
32,768, the reduction is around 48%.

When GOGC is set to the default value, the reductions are roughly 50%
and 46% for cache sizes of 16,384 and 32,768, respectively.

The "Heap in Use" column serves as a good indicator of memory
consumption and can assist in determining optimal cache sizes.

---

Proctree Stressor Benchmark CPU time (5 threads, 700000 iterations each)

| Method | New      | New (CPU %) | Old       | Old (CPU %) | Overhead |
|--------|----------|-------------|-----------|-------------|----------|
| Set    | 0.41 min | 0.74%       | < 0.01 hs | 0.87%       | -15.0%   |
| Get    | 0.04 min | 0.07%       | < 0.01 hs | 0.87%       | -92.0%   |
  • Loading branch information
geyslan committed Nov 11, 2024
1 parent b56b186 commit 215c94f
Show file tree
Hide file tree
Showing 6 changed files with 713 additions and 689 deletions.
Loading

0 comments on commit 215c94f

Please sign in to comment.