DataFusion repo got 40MB larger

[alamb]

Describe the bug

We accidentally have checked in a 40MB binary docs.tgz file in #10407

#10416 removed the file but it is still in git history

Thus the DataFusion repo is significantly larger than it used to be

To Reproduce

No response

Expected behavior

No response

Additional context

No response

[findepi]

That's unfortunate, but fixing this requires rewriting git history.
Is this issue about such a rewrite? or preventing similar mistake in the future?

[alamb]

That's unfortunate, but fixing this requires rewriting git history. Is this issue about such a rewrite? or preventing similar mistake in the future?

I suggest we use this particular issue for fixing the history -- I recommend a seprate ticket for ways to prevent a similar mistake in the future

[findepi]

if we want to fix this, this will require a force push to main branch and also doing something about tags that reference the commit that includes docs/docs.tgz file (39.0.0, 39.0.0-rc1, 40.0.0, 40.0.0-rc1).

Fixing main is simple (but painful)

i prepared a main branch copy with the docs archive removed (based on main as of today, commit 1331288), someone would need to push that into the main

git fetch https://github.com/findepi/datafusion findepi/main-without-docs
main_fixed=$(git rev-parse FETCH_HEAD)
current_main=133128840ca3dbea200dcfe84050cb7b82bf94a8 # commit on `main`
fork_point=$(git merge-base $main_fixed $current_main)

# verify
git show $main_fixed
git diff $current_main $main_fixed # same content

# verify history -- this should show `remove docs archive` commit omitted
git diff <(git log $fork_point..$current_main --format=%s) <(git log $fork_point..$main_fixed --format=%s)

# apply
git checkout main
git reset --hard $main_fixed

# apply commits from real current main
git fetch https://github.com/apache/datafusion main
git cherry-pick $current_main..FETCH_HEAD

# publish
git push https://github.com/apache/datafusion main:main --force-with-lease

once we do the above, then we need to delete (replace?) the release tags mentioned above

[alamb]

I think the branch protections would need to be updated temporarily too -- I don't think we can force push to main

I also think if we force push to main all outstanding PRs will become quite messed up until after a rebase

THank you @findepi for preparing this

@andygrove / @comphead / @jayzhan211 do you have any thoughts on this matter?

[comphead]

I think it was a precommit github hook checking large files, please allow me some time to investigate it.

https://github.com/pre-commit/pre-commit-hooks?tab=readme-ov-file#check-added-large-files

[findepi]

I also think if we force push to main all outstanding PRs will become quite messed up until after a rebase

that's correct.

also, editing release tags is something to think about.
if we re-use tag names, existing forks/clones will choke on this.

i'd be tempted not to solve this problem.

precommit github hook checking large files

is this something we can configure in the repo itself?
cause we should not rely on git commit hooks -- a contributor may not set them up.
we could have a CI check for this (like https://github.com/ppremk/lfs-warning , https://github.com/james-callahan/gha-prevent-big-files or something similar, perhaps our own)

[comphead]

Git action looks even more promising as you right, the precommit checks are not reliable as they running locally.
Imho I like the way https://github.com/james-callahan/gha-prevent-big-files it seems simpler for now and we have much control of it.
Would you like to experiment with this @findepi ?

[findepi]

@comphead sure
created #11508 with the check

[ozankabak]

Talking about the size and "cleanliness" of the repo, should we do a git gc? I find conflicting information on the Internet about whether GH does this automatically. In my experience GH doesn't seem to do this.

[findepi]

@ozankabak good question. git gc can remove only unreachable objects. Files/content that is part of repo history cannot be deleted without rewriting the history, so it's not something git gc could help with.

[ozankabak]

Yep, git gc and cleaning up history are different things, but I wanted bring this up as we make efforts into cleaning up the repo in general