Redesign questions

[grimmySwe]

@ingydotnet When I started to write some documentation and tried to explain how git-subrepo works I came to some insights. (I have discussed this a lot with @jrosdahl )

The main reason for these thoughts are that the most common use case where you repeatedly pull things and have some minor local change would be increasingly more difficult. Every time you pull you would find the common ancestor further back in history and you would have to solve the same conflicts over and over again. We already discussed in other issues that it would be nice if you could avoid having an extra branch if you want local changes.

• When performing a pull, it should be possible to use the pull reference in .gitrepo instead of the merge-base command. And instead of rebasing onto the new branch you should be able to simply apply the diff between the current state and the rebase parent. As the commits get squashed anyway.
  • Replace subrepo merge-base with simple use of .gitrepo.commit
  • Replace rebase with merge
  • Simplify subrepo branch

D(a) - E(a) - F(a) project   (commit(.gitrepo.commit))
a - b - c subrepo
$ git subrepo pull
* fetch
a - b - c subrepo/fetch
* branch
d - e - f subrepo/branch
* reparent f on a (see further down for reparenting tool)
a - b - c subrepo/fetch
  \
    f subrepo/branch
* merge, solve conflicts
a - b - c - g subrepo/fetch
 \         /
     f            subrepo/branch
* return to project
* write contents from g into subrepodir
* Update .gitrepo file
* commit
D(a) - E(a) - F(a) - G(g) project

Note that you if you really want to use rebase instead of merge you can do the reparenting on e instead and rebase the entire branch step by step. It will though add a little complexity as you would need to find out the first commit with the last .gitrepo.commit. I guess that one way would be to iterate from the HEAD and find the first one.

• For the push case it's a little bit trickier, in our use case it might be better to squash the push commit as well. Then you have the freedom of simply using the same simple logic of comparing the changes from the latest pull point and apply them to subrepo. If you keep the condition that you need to pull before you push it might even be simpler as all you need to do is create a new commit with the content of project/subrepo
  • Squash subrepo upstream
  • Simplify subrepo branch

D(a) - E(a) - F(a) project   (commit(.gitrepo.commit))
a subrepo
* fetch
a subrepo/fetch
* branch
d - e - f subrepo/branch
* checkout subrepo/fetch
* write contents of f 
* commit (it will though get a new commit hash)
a - f subrepo/fetch
* push
a - f subrepo

• If you really want to push all commits into the subrepo there are some strange things occuring. The commit sequence that appears in the subrepo is not linear, as the changes will occur in a rebased order including possible "pull" commits that reintroduces the changes already existing in the subrepo.

I found a tool for reparenting:
https://github.com/MarkLodato/git-reparent

When I think about it I guess that you could make the branch command work on only the last commit. That way you have constant speed. Current design will increase the time for each commit.

$ git checkout -b subrepo/branch
$ git filter-branch --subdirectory-filter subrepo -- HEAD^..HEAD

I haven't done any coding yet, all of this came up when I tried to describe how git-subrepo works.

[grimmySwe]

And yes I realize that much of this is returning to 0.2.3... solving the problems found then in another way.

Regarding the squash/no-squash it might be goo to have that optional on a subrepo basis. From my personal perspective I think that the squash method would be preferable in a corporate environment where you end up with huge amount of commits. But I do understand that in smaller repositories it might be beneficial to keep everything.

[grimmySwe]

One more thing, today there is a -u flag to subrepo push allowing you to update .gitrepo. I think that it might be a good idea to make this default behaviour. And in the case of squashing the push you get the same message on both sides.

$ git subrepo push -m "Adding feature X" bar

If you don't pass -m you could open up git_editor with the commit listing. This commit listing might need a previous push reference.

That way you could work by only pushing if no one else is working in the repo.

[grimmySwe]

more thinking...
If you decide to push and want all commits visible in subrepo, it might be useful to use the rebase --interactive style, so you can actually remove uneccessary commits/pulls.

[grimmySwe]

Tested some parent stuff and this seems to be working

$ git checkout master
$ git checkout -b subrepo/shared/branch
$ git filter-branch --subdirectory-filter shared -- HEAD^..HEAD
... # remove .gitrepo as well
$ git log --format=%T
<tree hash>
$ git commit-tree <tree hash> -p <.gitrepo.parent> -m "Some message"
<commit hash>
$ git reset --hard <commit hash>

Now we have a subrepo/shared/branch with our changes connected to the last pull parent. So from here we can do a git merge.

Note to self:
Test to create a new commit from subrepo/fetch based on project, then filter-branch.

[ingydotnet]

@grimmySwe, I'll need to pore over this more, later. One thing I don't quite
get is why you'd ever want to not squash.

As long as you can fetch the upstream commit in question, you always have the
history.

Educate me! :)

[grimmySwe]

@ingydotnet
To clearify this proposal, currently we have this situation:

• Pull
  • Uses merge-base to find equal tree hashes as common ancestor
  • Use rebase to apply changes from the subrepo
  • Creates a squashed commit in your project
  • Updates .gitrepo
• Push
  • You need to be up to date
  • Uses merge-base to find equal tree hashes as common ancestor
  • Use rebase to apply changes into subrepo
  • Commits are left as is, not squashed together
  • Can update .gitrepo with -u

Want I am suggesting is the following:

• Pull (Get latest changes from subrepo)
  • Use the .gitrepo.commit as common ancestor
  • Create subrepo/branch with .gitrepo.commit as parent and the content of project
  • Use merge between subrepo/fetch and subrepo/branch
  • Creates a squashed commit in your project (same as now)
  • Update .gitrepo.commit
• Push (put your state in subrepo)
  • You need to be up to date
  • Simply write your latest state to subrepo
  • Creates a squashed commit in the subrepo
  • Update .gitrepo.commit in project

Further extension to this would be to add a --no-squash flag to push:

• Push --no-squash (put your commits into the subrepo)
  • You need to be up to date
  • Create a correct graph in the subrepo of your commits (so you get real merge commits inside the subrepo) from the .gitrepo.last_push
  • Update .gitrepo.commit and a new .gitrepo.last_push

Even more:
Add a subrepo configuration, so that you can set a squash/no-squash setting in the actual subrepo so that the default behaviour is set for a subrepo.

[grimmySwe]

Also consider #144, using filter-branch on an unknown amount of commits might be very slow.

[grimmySwe]

Regarding the push case without squashing: If you store the last push in .gitrepo.

First perform a parent-filter subdirectory from the latest push to the current position. Now you can step for each commit, look into which commit it points to in .gitrepo and when that changes add it to the parent list. Finally perform the merge and the push the entire tree into subrepo. That way you would get a complete version in subrepo, containing actual merge commits.
Finally you update the local .gitrepo with new last push and latest commit.

[grimmySwe]

@ingydotnet @jrosdahl I have created a first attempt at this (Branch issue/142) and need some feedback.
Note: I haven't bothered about the unittest/documentation yet

Status:

• both push/pull are squashed.
• You can't push unless you have recently pulled.
• When you create the subrepo/branch it creates a new node with the .gitrepo.commit as parent.
• Merge is used instead of rebase
• subrepo push will also create a local commit updating the .gitrepo file

Feedback needed:
Currently the squashed push message is simply PLACEHOLDER... What would you consider the best approach:

1. A generic message like the one used in pull/commit
2. User defined with -m/editor opens if -m is missing

I guess the ultimate would be some kind of hint regarding the content, but this is a little bit tricky as you don't really now what the content is right now.

The immediate benefits of this solution:

• You can have a local change and still pull in changes without having to solve the same conflicts over and over again.
• You can work in push mode, performing consecutive pushes as long as no one else pushes.
• "Constant" time operation for the branch creation. You always use the .gitrepo.commit pointer so you don't have to iterate the tree.

Drawbacks:

• Currently push commits are squashed
• Not all commit messages are relevant

[robe070]

For our use case we absolutely must not squash push commits. The real commit messages must be retained in the subrepo history. So I would like to see an option for this and that not squashing is the default as it is the current behaviour.

[grimmySwe]

@robe070 I am working on the next step of this, where you don't get squashed pushes (unless wanted) and where you hopefully get a logically correct tree in in the subrepo. One of the drawbacks with the current implementation is that when you perform rebases they will end up in a somewhat unlogical order, so the history in the subrepo might be strange.

Not squashing actually solves another issue that you don't have to generate the push commit messages.