Skip holes #579
Closed
Skip holes #579
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This relies on SEEK_HOLE and SEEK_DATA support, which is present in Linux, FreeBSD, Solaris and others, though not technically a POSIX standard yet.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi and thanks for making "ag", one of the most invaluable tools that I use daily!
Previously "ag -u asdfasdfasdfadsf /root_partition" was taking nearly ten minutes, due to gobblying gigabytes of zeros in sparse files, notably /var/log/lastlog, which is nominally 400+ GB on my system but actually occupies less than 100 KB
The behavior witnessed is the worker thread pegging its core to 100%, as well as the kernel swapper thread getting its core to 40% dutifully discarding those pages of zeros as ag faults in ever more of them.
This adds a --skip-holes option that jumps over holes in sparse files, using the SEEK_HOLE and SEEK_DATA API described here: https://blogs.oracle.com/bonwick/entry/seek_hole_and_seek_data (and in the years since, implemented on more OSes than Solaris)
Now with --skip-holes I can do ag -u of my root partition in something like 1:30 instead of 9:30, and ag -u of /var/log/lastlog takes seconds.