Using --refs to limit file removal to later parts of history doesn’t work

[huftis]

According to the documentation, we can use the --refs argument to limit rewriting to only the later part of the history. However, removing files from part of the history (up to and including the last commit) using --refs doesn’t seem to work.

Here’s a very simple example with three commits and three files. The file foo.txt is present in all three commits, bar.txt in the last two commits and baz in only the last commit:

First commit:
foo.txt

Second commit:
foo.txt
bar.txt

Third commit:
foo.txt
bar.txt
baz.txt

I want to delete the file foo.txt, but only from the last two commits. Based on the documentation, I would expect the following to work:

git-filter-repo --invert-paths --path foo.txt --force --refs HEAD~1..HEAD

But it doesn’t seem to have any effect. Reproducible example:

mkdir gfr-test
cd gfr-test
git init
echo foo > foo.txt
git add foo.txt
git commit -m "Add foo"
echo foo2 >> foo.txt
echo bar > bar.txt
git add foo.txt bar.txt
git commit -m "Add bar, modify foo"
echo baz > baz.txt
git add baz.txt
git commit -m "Add baz"

git-filter-repo --invert-paths --path foo.txt --force --refs HEAD~1..HEAD

Nothing seems to have changed:

$ git log --name-status
commit 67e998bd460feb06795e392e535a42af84baebc9 (HEAD -> main, origin/main, origin/HEAD)
Author: Nomen Nescio <nobody@example.com>
Date:   Sat Mar 29 18:00:52 2025 +0100

    Add baz

A       baz.txt

commit 476e3a829652c6bb89c19a32d92172ca2a982986
Author: Nomen Nescio <nobody@example.com>
Date:   Sat Mar 29 18:00:52 2025 +0100

    Add bar, modify foo

A       bar.txt
M       foo.txt

commit 3dbbc2c33bc4eb550f51f9df39695ca1f0b1f599
Author: Nomen Nescio <nobody@example.com>
Date:   Sat Mar 29 18:00:52 2025 +0100

    Add foo

A       foo.txt

The commits contain the same files as before:

$ git ls-tree HEAD
100644 blob 5716ca5987cbf97d6bb54920bea6adde242d87e6    bar.txt
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca    baz.txt
100644 blob b210800439ffe3f2db0d47d9aab1969b38a770a5    foo.txt
$ git ls-tree HEAD~1
100644 blob 5716ca5987cbf97d6bb54920bea6adde242d87e6    bar.txt
100644 blob b210800439ffe3f2db0d47d9aab1969b38a770a5    foo.txt
$ git ls-tree HEAD~2
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99    foo.txt

Expected results:

$ git log --name-status
commit 67e998bd460feb06795e392e535a42af84baebc9 (HEAD -> main, origin/main, origin/HEAD)
Author: Nomen Nescio <nobody@example.com>
Date:   Sat Mar 29 18:00:52 2025 +0100

    Add baz

A       baz.txt

commit 476e3a829652c6bb89c19a32d92172ca2a982986
Author: Nomen Nescio <nobody@example.com>
Date:   Sat Mar 29 18:00:52 2025 +0100

    Add bar, modify foo

A       bar.txt
D       foo.txt

commit 3dbbc2c33bc4eb550f51f9df39695ca1f0b1f599
Author: Nomen Nescio <nobody@example.com>
Date:   Sat Mar 29 18:00:52 2025 +0100

    Add foo

A       foo.txt

$ git ls-tree HEAD
100644 blob 5716ca5987cbf97d6bb54920bea6adde242d87e6    bar.txt
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca    baz.txt
$ git ls-tree HEAD~1
100644 blob 5716ca5987cbf97d6bb54920bea6adde242d87e6    bar.txt
$ git ls-tree HEAD~2
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99    foo.txt

If I leave out the --refs argument, so that the removal is not limited to the later part of the history, everything is working fine (foo.txt is removed from all commits, and the first commit, which is now empty, is completely removed). (But if I use --refs HEAD~2..HEAD, which should in theory also affect the entire history, nothing happens.)

[azabost]

I think I'm facing the same issue right now. Trying to rename a directory like this:

git filter-repo --refs <old_commit>..<latest_commit> --path-rename <old_dir_name>:<new_dir_name>

It doesn't seem to do anything, the history is unchanged (both the content and the SHA-1's)

[newren]

I want to delete the file foo.txt, but only from the last two commits. Based on the documentation, I would expect the following to work:

git-filter-repo --invert-paths --path foo.txt --force --refs HEAD~1..HEAD

Multiple errors in understanding here:

• HEAD~1..HEAD only specifies one commit (the latest), not the last two commits.
• filter-repo is fast because it operates on the fast-export/fast-import stream, which only includes the files that have been modified in a given commit. As such, filtering out paths but only starting at some commit means you only remove the changes to those paths starting at that commit. If you start at the root commit, filtering out all changes to a certain file means you also get rid of the introduction of that file. If you start at some other commit, then you just remove all the changes since whatever commit.

The combination explains why this makes no change to your repository -- the final commit made no changes to foo.txt, so trying to discard the changes made to foo.txt in the final commit is a no-op.

But it doesn’t seem to have any effect. Reproducible example:

mkdir gfr-test
cd gfr-test
git init
echo foo > foo.txt
git add foo.txt
git commit -m "Add foo"
echo foo2 >> foo.txt
echo bar > bar.txt
git add foo.txt bar.txt
git commit -m "Add bar, modify foo"
echo baz > baz.txt
git add baz.txt
git commit -m "Add baz"

git-filter-repo --invert-paths --path foo.txt --force --refs HEAD~1..HEAD

Thanks for the well-written simple example.

The commits contain the same files as before:

$ git ls-tree HEAD
100644 blob 5716ca5987cbf97d6bb54920bea6adde242d87e6 bar.txt
100644 blob 76018072e09c5d31c8c6e3113b8aa0fe625195ca baz.txt
100644 blob b210800439ffe3f2db0d47d9aab1969b38a770a5 foo.txt
$ git ls-tree HEAD1
100644 blob 5716ca5987cbf97d6bb54920bea6adde242d87e6 bar.txt
100644 blob b210800439ffe3f2db0d47d9aab1969b38a770a5 foo.txt
$ git ls-tree HEAD2
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99 foo.txt

Yes, as I'd expect.

Expected results:
...
commit 476e3a829652c6bb89c19a32d92172ca2a982986
Author: Nomen Nescio nobody@example.com
Date: Sat Mar 29 18:00:52 2025 +0100

Add bar, modify foo

A bar.txt
D foo.txt
...

In order to get this in combination with partial history rewrite, you'd have to use a --commit-callback and insert a deletion FileChange for foo.txt. Something like (warning: untested -- typing up off the top of my head):

git filter-repo --commit-callback '
    if commit.original_id = b'5ca1fo1d5ca1fo1d5ca1fo1d5ca1fo1d5ca1fo1d':
        commit.file_change.append(FileChange(b'D', b'foo.txt'))
'

You could take this commit-callback and use together with your --refs, --path, and --invert-paths flags. In that case, filtering out foo.txt would still be wanted to prevent later commits from re-introducing the file.

(But if I use --refs HEAD~2..HEAD, which should in theory also affect the entire history, nothing happens.)

Two errors with this statement here:

1. No, that should not affect the entire history; this only affects the last two commits. In git, A..B excludes A (and all commits reachable from A).

2. I don't think you looked closely enough, this command DOES change history; running it on your example setup:

$ git log --raw --oneline
6946d08 (HEAD -> master) Add baz
:000000 100644 0000000 7601807 A        baz.txt
5d649aa Add bar, modify foo
:000000 100644 0000000 5716ca5 A        bar.txt
:100644 100644 257cc56 b210800 M        foo.txt
7c48a70 Add foo
:000000 100644 0000000 257cc56 A        foo.txt
$ git-filter-repo --invert-paths --path foo.txt --force --refs HEAD~2..HEAD
Parsed 2 commits
New history written in 0.02 seconds...
HEAD is now at e9865ca Add baz
Completely finished after 0.03 seconds.
$ git log --raw --oneline
e9865ca (HEAD -> master) Add baz
:000000 100644 0000000 7601807 A        baz.txt
bcbd19f Add bar, modify foo
:000000 100644 0000000 5716ca5 A        bar.txt
7c48a70 Add foo
:000000 100644 0000000 257cc56 A        foo.txt

...notice that before filtering, the middle commit modified both bar.txt and foo.txt, while after filtering, the middle commit only modified bar.txt.

[newren]

I think I'm facing the same issue right now. Trying to rename a directory like this:

git filter-repo --refs <old_commit>..<latest_commit> --path-rename <old_dir_name>:<new_dir_name>

It doesn't seem to do anything, the history is unchanged (both the content and the SHA-1's)

This looks like a different misunderstanding and issue than above. Any command including --refs <old_commit>..<latest_commit> where <latest_commit> is a git commit id rather than a branch name, is basically a no-op. It's not quite a no-op -- it writes a bunch of new commits that nothing refers to into your object store which will then later need to be garbage collected. In order for --refs to do anything useful, the right hand side of any A..B range has to be the name of a ref so that the ref will get updated. (This is tangentially address in the last paragraph of https://github.com/newren/git-filter-repo/blob/main/Documentation/FAQ.md#why-did-git-filter-repo-rewrite-more-commit-hashes-than-i-expected)

[azabost]

I think I'm facing the same issue right now. Trying to rename a directory like this:
git filter-repo --refs <old_commit>..<latest_commit> --path-rename <old_dir_name>:<new_dir_name>
It doesn't seem to do anything, the history is unchanged (both the content and the SHA-1's)

This looks like a different misunderstanding and issue than above. Any command including --refs <old_commit>..<latest_commit> where <latest_commit> is a git commit id rather than a branch name, is basically a no-op. It's not quite a no-op -- it writes a bunch of new commits that nothing refers to into your object store which will then later need to be garbage collected. In order for --refs to do anything useful, the right hand side of any A..B range has to be the name of a ref so that the ref will get updated. (This is tangentially address in the last paragraph of https://github.com/newren/git-filter-repo/blob/main/Documentation/FAQ.md#why-did-git-filter-repo-rewrite-more-commit-hashes-than-i-expected)

Thanks @newren . Clearly, I misunderstood the documentation (or rather that ref != revision).