Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BANIDX facility to speed up BAN loading #4261

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

nigoroll
Copy link
Member

@nigoroll nigoroll commented Feb 5, 2025

When loading a cache with a relevant number of BANs from a persistent storage, BAN_FindBan() needs to called for every object to determine the struct ban which the in-core object is to point to based on the persisted ban time.

This is highly inefficient because BAN_FindBAN() conducts a linear search of the ban list, for which we would expect about half the bans to need inspection if objects were distributed equally across all bans. But for a long BAN list, most objects will hang near the bottom, so chances are high that we actually need to traverse nearly all of the ban list for each loaded object.

So this screams for turning the ban list into a different data structure, like a binary tree.

Yet once loading of objects is complete, the ban list as we have it is optimal: All operations on the ban list need to traverse it anyway and in many cases this traversal can happen unlocked. So we really would not want to change the structure of the ban list.

This patch suggests to add a search tree as an additional data structure, which only exists during cache load (while there are ban_holds). In particular, the interface is made such that the only relevant change to the ban code is to provide a hint for the linear search to start at: VTAILQ_FOREACH(ban, ...) gets turned into

	ban = BANIDX_lookup(timestamp);
	VTAILQ_FOREACH_FROM(ban, ...);

and that's it.

In particular, this patch implies no change at all to setups which do not use persistent storage, so the risk is considered very low.

The speed up is impressive:

The following tests were conducted with a SLASH/fellow storage with 7689 bans. For each test round, the storage (residing on a ZFS volume) was rolled back to the same state, basically like so:

	zfs rollback int21/slash/10g@reproduce
	varnishd ... -sfellow=fellow,/dev/zvol/int21/slash/10g,10GB,1g,32k ...

When the cache was loaded, varnishd was terminated and the test repeated.

Six runs were conducted with Varnish-Cache master as of 749a2c3, resulting in load times in the order of 21 seconds:

        0 Storage        - fellow fellow: 289627 resurrected in 28.477966s (10235.386784/s), 1856 already expired
        0 Storage        - fellow fellow: 289627 resurrected in 26.170669s (11137.774192/s), 1856 already expired
        0 Storage        - fellow fellow: 289627 resurrected in 24.331572s (11979.620642/s), 1856 already expired
        0 Storage        - fellow fellow: 289626 resurrected in 21.455001s (13585.783644/s), 1857 already expired
        0 Storage        - fellow fellow: 289626 resurrected in 21.340564s (13658.635888/s), 1857 already expired
        0 Storage        - fellow fellow: 289626 resurrected in 21.568622s (13514.215014/s), 1857 already expired

(note: the change of the number of cache objects is because wall clock time has passed the expiry time of one object)

With this patch, load times were reduced by a factor >15

        0 Storage        - fellow fellow: 289625 resurrected in 1.293932s (225269.103288/s), 1858 already expired
        0 Storage        - fellow fellow: 289625 resurrected in 1.416040s (205843.804318/s), 1858 already expired
        0 Storage        - fellow fellow: 289625 resurrected in 1.341485s (217283.788555/s), 1858 already expired

Naturally, the benefits are even more relevant with a higher number of bans.

@nigoroll
Copy link
Member Author

@mbgrydeland I would appreciate your feedback.

When loading a cache with a relevant number of BANs from a persistent storage,
BAN_FindBan() needs to called for every object to determine the struct ban which
the in-core object is to point to based on the persisted ban time.

This is highly inefficient because BAN_FindBAN() conducts a linear search of the
ban list, for which we would expect about half the bans to need inspection if
objects were distributed equally across all bans. But for a long BAN list, most
objects will hang near the bottom, so chances are high that we actually need to
traverse nearly _all_ of the ban list for each loaded object.

So this screams for turning the ban list into a different data structure, like a
binary tree.

Yet once loading of objects is complete, the ban list as we have it is optimal:
All operations on the ban list need to traverse it anyway and in many cases this
traversal can happen unlocked. So we really would not want to change the
structure of the ban list.

This patch suggests to add a search tree as an _additional_ data structure,
which only exists during cache load (while there are ban_holds). In particular,
the interface is made such that the only relevant change to the ban code is to
provide a hint for the linear search to start at: VTAILQ_FOREACH(ban, ...) gets
turned into

	ban = BANIDX_lookup(timestamp);
	VTAILQ_FOREACH_FROM(ban, ...);

and that's it.

In particular, this patch implies no change at all to setups which do not use
persistent storage, so the risk is considered very low.

The speed up is impressive:

The following tests were conducted with a SLASH/fellow storage with 7689 bans.
For each test round, the storage (residing on a ZFS volume) was rolled back to
the same state, basically like so:

	zfs rollback int21/slash/10g@reproduce
	varnishd ... -sfellow=fellow,/dev/zvol/int21/slash/10g,10GB,1g,32k ...

When the cache was loaded, varnishd was terminated and the test repeated.

Six runs were conducted with Varnish-Cache master as of
749a2c3, resulting in load times in the order
of 21 seconds:

        0 Storage        - fellow fellow: 289627 resurrected in 28.477966s (10235.386784/s), 1856 already expired
        0 Storage        - fellow fellow: 289627 resurrected in 26.170669s (11137.774192/s), 1856 already expired
        0 Storage        - fellow fellow: 289627 resurrected in 24.331572s (11979.620642/s), 1856 already expired
        0 Storage        - fellow fellow: 289626 resurrected in 21.455001s (13585.783644/s), 1857 already expired
        0 Storage        - fellow fellow: 289626 resurrected in 21.340564s (13658.635888/s), 1857 already expired
        0 Storage        - fellow fellow: 289626 resurrected in 21.568622s (13514.215014/s), 1857 already expired

(note: the change of the number of cache objects is because wall clock time has
 passed the expiry time of one object)

With this patch, load times were reduced by a factor >15

        0 Storage        - fellow fellow: 289625 resurrected in 1.293932s (225269.103288/s), 1858 already expired
        0 Storage        - fellow fellow: 289625 resurrected in 1.416040s (205843.804318/s), 1858 already expired
        0 Storage        - fellow fellow: 289625 resurrected in 1.341485s (217283.788555/s), 1858 already expired

Naturally, the benefits are even more relevant with a higher number of bans.
@nigoroll nigoroll added a=need bugwash a=last call MR will be merged if no veto within 7 days labels Mar 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a=last call MR will be merged if no veto within 7 days a=need bugwash
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant