Support clean / smudge filtering

[krlvi]

[krlvi]

libgit has it libgit2/libgit2#1473

[droidicus]

The libgit2 site is down for me at the moment, here is the wayback from last week of the API docs: https://web.archive.org/web/20240212173946/https://libgit2.org/libgit2/#HEAD/group/filter

[ethomson]

Yeah, I broke the DNS for libgit2.org. 😓 It should be back now: https://libgit2.org/libgit2/#HEAD/group/filter

We recently taught libgit2 to invoke external processes - like ssh. The other use for this is, obviously, external filter drivers (like LFS). I have a branch somewhere that I can push up if this would be helpful for us to iterate on together.

[schacon]

Yeah, I think the libgit2 filter functions aren't implemented in git2-rs yet. It's probably not super common to have much other than crlf and lfs filters, so we could special case those, OR, we could fully implement smudge/clean stuff with custom commands. [heavy sigh]. I'm not sure how much work that is. Maybe we could start with the simple, common stuff and see what that looks like first.

[ethomson]

Do you want to have your own filters that you customize? Or do you just want to support filter.*.smudge and filter.*.clean and have them Just Work?

If it's the former, yeah, that's hard and requires git2-rs. If it's the latter, let's let libgit2 just Run The Things For You.

I imagine some sort of setting like git_libgit2_set_opt(GO_RUN_FILTERS_FOR_ME, 1) and then voila.

I'll see if I can dig up my branch — most of the hard bits were actually here: libgit2/libgit2#6617

[droidicus]

@ethomson I had originally talked to @krlvi on discord about this. My original question was:

I use both Python and iPython Notebooks in my projects, and use https://pypi.org/project/nbstripout/0.2.5/ installed as a filter in my repos to prevent the runtime changes from affecting the notebook files. However, it appears that the changes typically filtered by git are showing up in git-butler, and this is causing failures in branches where I have changed *.ipynb files.
Does git-butler currently support git clean/smudge filters, or is this broken in some other way?

I don't have any need to create new filters, I just need to run the filters that already exist in the repository with git on the CLI. Currently this configuration results in broken branches, so I cannot use git-butler for my use case.

[tfylling]

I guess the lack of support for smudge filters is the reason why all the files encrypted with git-crypt are suddenly listed in a branch? This only started happening today though, after I updated to 0.12.2, and it creates a mess with several hundred files obscuring any real changes. So far the only workaround I've come up with is to just move all these files to a separate branch and keep another branch as the default, but they're still an eyesore, and I need to keep adding a new default branch if I delete the second last one.

[Byron]

Just so that I don't forget myself, let's state the upcoming release will already be able to create commits with clean-filters applied, i.e. commits created with LFS files will look correct. However, whenever worktree operations happen and affect LFS files, these would be checked out incorrectly as long as git2 is still used for that.
gitoxide would need to learn how to do worktree resets, which at this point can 'just' be implemented, once there is time.

However, that's probably not going to happen to soon, and there will be many upcoming releases without proper support because of it.