Preparing for the merge

[snowleopard]

So, it looks like Hadrian is finally getting merged! 🎉

The plan is to merge Hadrian as is, without yet recommending GHC developers to switch to it, but already requiring all GHC patches to be buildable both by Make and by Hadrian.

Let's use this issue to keep track of what needs to be done.

• Fix all references to GitHub issues and pull requests in the source from #N to concrete GitHub links to avoid confusion with GHC tickets.
• Prepare the patch, using git subtree to keep Hadrian repository on GitHub.
• Update the README.

Anything else?

In my view, the two most important requirements in the long term are:

1. Preserving the commit/issue/pull-request history. A GHC developer fighting a strange build failure should be able to find a relevant discussion not only now but in 5 years from now. This may be solved via documentation, i.e. gradually moving all discussions from GitHub to docs/comments. That's a lot of hard work (compared to simply keeping Hadrian's repository here on GitHub).

2. Making it convenient for GHC developers to work on Hadrian. To me, git submodules are not convenient at all, but maybe there are just no other options given the requirement (1). Is git subtree a solution?

[izgzhen]

If using git subtree, where the main development will happen? In this repo, or in the GHC tree?

[angerman]

As I've raised the same question again. I believe this should be a submodule while development continues on GitHub. And then become a subtree, once development will shift over to phabricator.

[snowleopard]

@bgamari Can you comment on why you think git subtree is the way to go?

[snowleopard]

My understanding of the difference between subtree and submodule is:

• subtree will keep a copy (a snapshot) of Hadrian's repo in the GHC tree. To update the copy we can subtree pull from Hadrian's repo, probably squashing the commits, and push them to GHC. It is also possible to push changes made to Hadrian in GHC tree to Hadrian's repo.
• submodule will keep a link to a specific commit in Hadrian's repo. To update GHC's version of Hadrian, one needs to change the link to point to a fresher commit in Hadrian's repo. With this approach the only way to change Hadrian is to do it in Hadrian's repo.

@izgzhen Answering your question, I think the main development will at some point shift from GitHub to the GHC tree, but we will be able accept patches in the GHC tree at any point and push these changes to GitHub. This will require some extra work from us, but it's possible using subtree route.

@angerman You say you "believe" we should use git submodule instead of git subtree. Can you be more specific? It looks to me that git subtree is strictly more powerful, but perhaps I'm missing something.

[snowleopard]

P.S.: I just noticed there is a continued discussion on the ghc-devs mailing list with detailed comments from @hvr and @angerman. I'd prefer to keep the discussion here, as email is not the best medium.

[angerman]

Just for reference, the email conversation @snowleopard is referring to is the one starting here: https://mail.haskell.org/pipermail/ghc-devs/2017-October/014890.html

@snowleopard my initial impression of git subtree was likely flawed and it doesn't really embed one repo into another. However, I'll say this again. For consistency I would very much prefer git submodule over git subtree. As every other external reference in the ghc tree is a submodule.

Let me just add, that with the git submodule, you will also be able to work on hadrian in ghc. There is effectively the hadrian repository inside of ghc, and you can push from there as well. (Of course there is the issue with the detached head; which needs some getting used to). I do see however that due to some technicalities this might only work if hadrian was moved into the ghc organization? as the submodule would have to point to GitHub.com/ghc/hadrian, instead of GitHub.com/snowleopard/hadrian? But I'm certain @hvr will be able to explain the necessary requirements better here. Not sure if the subtree would impose those technicalities as well.

[snowleopard]

However, I'll say this again. For consistency I would very much prefer git submodule over git subtree. As every other external reference in the ghc tree is a submodule.

@angerman Yes, this is a very good point. Indeed, git submodules are familiar to all GHC developers.

Let me just add, that with the git submodule, you will also be able to work on hadrian in ghc. There is effectively the hadrian repository inside of ghc, and you can push from there as well.

Aha, I didn't realise this!

I do see however that due to some technicalities this might only work if hadrian was moved into the ghc organization? as the submodule would have to point to GitHub.com/ghc/hadrian, instead of GitHub.com/snowleopard/hadrian?

I have no problems with moving this repository into the ghc organisation. As far as I understand this should not affect commits/issues/PRs. In fact, being part of the ghc organisation makes a lot of sense. I should have probably started the repo there in the first place. I've added a corresponding todo item above.

[bgamari]

Indeed, git submodules are familiar to all GHC developers.

Yes, but submodules and subtrees are two quite different solutions addressing rather different use-cases; they are not interchangeable. The former is essentially a pointer to a commit in an external repository. As far as the history of the containing repository is concerned the submodule has no "structure". The submodule is just a file with a commit SHA in it. We indeed use submodules for tracking third party libraries since we don't want their history to pollute the ghc tree.

Creating a subtree, on the other hand, essentially amounts to merging one repository's history into another. This is ultimately the state we want Hadrian to end up in and therefore I think this is the state where I think we should start the merge.

Starting with a submodule would require a fair amount of administrative work (since we have to setup mirrors) and yet all of this work would be for naught in the end; we would eventually have to rip out of the submodule and replace it with... a subtree. I have a hard time seeing the sense in this.

If we are going to merge, let's do so in a way that won't need to be reverted later on. With the subtree approach development can continue here and GHC can easily pull over changes on a regular basis (even once an hour, if we want). Updating is merely a git subtree pull hadrian master away.

[snowleopard]

@bgamari I see you point. How will this look from the point of view of a GHC developer? We'll need to provide some guidance in the README, so people are not lost. What about the following?

• If you need to build GHC with Hadrian, run ./build -j from the GHC root. (This assumes we create top-level scripts build.sh and build.bat that simply redirect to appropriate scripts in Hadrian subdirectory.) For this very common use-case there is no need to know about subtree at all.
• To bring Hadrian up-to-date: git subtree pull hadrian master.
• To make a change to Hadrian: edit the source file, and do git commit and git subtree push ??? -- this bit is unclear to me. What is the right process?

Maybe this will also convince @angerman that there is nothing too difficult about the subtree workflow.

[hvr]

Fwiw, I have the following in my ~/.gitconfig:

[alias]
	pullall = "!f(){ git pull \"$@\" && git submodule update --init --recursive; }; f"

so I can git pullall or git pullall --rebase and it does the right thing; how exactly would I need to change that to have the same "rebasing" pull for a subtree?

[Ericson2314]

@hvr That will continue to work!

The git subtree pull hadrian master people mention is analogous to changing the commit hash of a submodule:

pushd "$foo" \
 && git checkout origin master \
 && git pull && popd \
 && git add "$foo" \
 && git commit -m "update $foo submodule"

a mouthful which is not part of your alias.

[Ericson2314]

For comparison, the Rust people are going with the subtree approach as we speak with their core language interpreter:

1. Prep miri repository for rustc merger rust-lang/miri#258 in the interpreter repo
2. Merge miri into librustc_mir rust-lang/rust#43340 in the main repo

[hvr]

@Ericson2314 do I understand it right that git subtree pull hadrian master is the equivalent of git submodule update --remote ./hadrian for subtrees?

[angerman]

Can we maybe start with the "small" and easy things? E.g. move this to github.com/ghc? @hvr, @bgamari, who can give relevant permissions?