Repository navigation
Discussion: What approach to use for migrating extensions #13
Description
Activity
I have a running version here: https://github.com/NickSdot/php__doc-extensions
Includes all cleanups, and additionally fixer runs per extension. You should be able to just use that repo and force push. Potential further work then could just happen on top of the current state. 🕺
The three existing commits were cherry picked on top of the history:
- NickSdot/php__doc-extensions@b6763ca
- NickSdot/php__doc-extensions@91362ad
- NickSdot/php__doc-extensions@03a8817
I added five more commits to get things cleaned up and working:
- NickSdot/php__doc-extensions@91cc47f
- NickSdot/php__doc-extensions@4697387
- NickSdot/php__doc-extensions@8b796d1
- NickSdot/php__doc-extensions@4b78bcc
- NickSdot/php__doc-extensions@c4a27b7
All other commits are the (hand reviewed) fixer runs per extension. Two of it needed intervention.
I went multiple times back and forth whether to wait for the final extension list, or migrate all mentioned in the RFC. I eventually settled on the latter. In case some must not be migrated, they can be deleted in a separate follow up PR per removal. This way we make sure we do not "betray" contributors of their hard work and commits -- even if an extension will be deleted eventually, the history would remain in tact so historical contributors are treated respectfully; I think it's important. ❤️
The next step, when this is accepted, could be that we prepare the inverse of this one. As in,
doc-ennow contains things that are no longer needed there because it is in relevant only indoc-extensions.All in bulk with shared git commit history
This. In fact, migrate all
reference/history, and then remove folders that will keep existing in the main manual. Each extension that is decided to keep in the main manual are deleted here, and extension that is decided to be moved is marked on doc-en asMOVING_TO_DOC_EXTENSIONS_DO_NOT_CHANGE_HERE.md, directing efforts into making this repo to build.At first nothing in here may build, but that is ok, the main manual still exists. Two known efforts into splitting the manual shows that the problem is manageable, so this duplicated state will not last. And having more people caring for this manual to build will accelerate the process.
Then, after both manuals are online remotion efforts on doc-en may start, removing extensions by unit or batch, fixing any problem that shows. There will be "jump" links to be created in doc-en to doc-extensions, as was necessary in the opposite direction.
Unfortunately, I do not finished my long project of eliminating SGML/DTD partes of manual before the split, but after doc-base 321 lands, and with help of
doc-base/blob/master/scripts/dtdent-conv.php, the doc-extension may start without any DTD files.If things get into a grind, I may try to accelerate the "userland XML assembler", that will help debug DTD entity reference usage. I think that @jordikroon already has something similar for docbook-cs.
I would suggest
- fork doc-en
- delete the stuff that isn't needed
- update for the new structure
and that way we keep all of the git history. thoughts?
and that way we keep all of the git history. thoughts?
One downside of this is that you keep unaffected history (and blobs) in both repos instead of having two cleanly split ones that only have the contents and history of what they still represent.
Good point, we could use
git filter-repo(or filter branch) instead to keep contributor history, those hashs would changeGood point, we could use
git filter-repo(or filter branch) instead to keep contributor history, those hashs would changeYeah, thats's what I did in the repo that I linked in the answer above. It's basically done -- if everyone agrees that the history should be preserved; which I am personally strongly in favour of.
The old commits start at this page (all following were me running the fixers on it):
https://github.com/NickSdot/php__doc-extensions/commits/main/?after=a2e3be97acafedf9227c509e3059f2e4787e19d3+69As long as the commit-credit history is preserved, I don't think the details of duplication or hashes or whatnot matter. Cloning and then using filter-branch/filter-repo seems like a fine approach.
Preserved hashes would by nice, if translations have a chance to come back.
Preserved hashes would by nice, if translations have a chance to come back.
The RFC decided to not carry over translations.
Thanks everyone for the input. It's been 2 weeks since the last reply, and the feedback has resulted in a concrete plan:
Phase 1: Finalising the extension list, see issue #12 which is planned to be locked on Tuesday.
With an immediate follow up to add aMOVING_TO_DOC_EXTENSIONS_DO_NOT_CHANGE_HERE.mdto direct any effort here. This way doc-en can't drift underneath us while the migration is in progress.
Phase 2: Extract the extensions from doc-en with git filter-repo, all in one pass:
git clone git@github.com:php/doc-en.git import && cd import git filter-repo --paths-from-file ../paths-to-migrate.txt # see note belowPush the result to this repo. The few existing commits here (structure + README) get replayed on top of the filtered history, followed by a one-time force-push of main.
Author/committer identities and dates are preserved, commits that touched multiple extensions survive as single commits. Hashes will change, but as concluded above: credit is the requirement, hashes are not.
Phase 3: Make it build. It's fine if nothing builds at first. The main work is converting DTD entities to XML entities, which is now unblocked since php/doc-base#321 landed, using doc-base/scripts/dtdent-conv.php. Goal: green CI on make.
Phase 4: Docbook-cs sweep.
docbook-cs --fix .over the repo. Which will fix all imperfections that currently exist in the migrated extensions.
Phase 5: doc-en cleanup. Removing the migrated extensions from doc-en per batch, closing open doc-en PRs with a pointer and transferring issues where feasible.
Phase 6: Publish and add redirects in coordination with the Infrastructure team / Derick.
Note: One caveat based on the current extension list. filter-repo follows paths, not renames, and there have been a few restructures over the course of 25 years. E.g. the 2002 restructure moved the old
functions/<ext>.xmlchapters intoreference/<ext>/. The list below contains these older paths. The final result should stay the same.https://gist.github.com/jordikroon/3ed74994513e5ce6574d346bc51333c6
Regarding translations. The hashes will change, but the history will not. If we were to support translations, work would have to be done regardless, and it would still be able to restore that history one way or another and update the hashes to reflect the new ones.
If you have any further comments, please drop. We could start with the migration once the extensions are finalises.
Author/committer identities and dates are preserved, commits that touched multiple extensions survive as single commits. Hashes will change, but as concluded above: credit is the requirement, hashes are not.
Dates are almost distinct, so old-hash-to-new-hash mappings can be almost perfectly done, on this field alone:
$ git log --name-only | grep -F 'Date:' > dates $ wc dates 31400 219798 1215718 dates $ cat dates | sort | uniq | wc 31365 219553 1214356Phase 5: doc-en cleanup. Removing the migrated extensions from doc-en per batch, closing open doc-en PRs with a pointer and transferring issues where feasible.
Phase 6: Publish and add redirects in coordination with the Infrastructure team / Derick.
Possibly revert the order here. First publish and start redirections, then remove extensions from the language manual.
Regarding translations. The hashes will change, but the history will not. If we were to support translations, work would have to be done regardless, and it would still be able to restore that history one way or another and update the hashes to reflect the new ones.
Per above, this possible work can be done almost by Date: field alone.
If you have any further comments, please drop. We could start with the migration once the extensions are finalises.
As a Step 2.5, please add in the README here a copy of these steps, while also noting the date and commit hash of the last push of filtered commits. Even if this info is later removed, only to exist in this fork history.
Also, take note and annotate here the date and commit hash of creation of all
MOVING_TO_DOC_EXTENSIONS_DO_NOT_CHANGE_HERE.mdin doc-en, as this file will not be created HERE.Or to have a very clean split marker, first create
MOVING_TO_DOC_EXTENSIONS_DO_NOT_CHANGE_HERE.mdand then run the clone-filter-push, so it would be very intuitive where any posterior change in doc-en would be clearly discarded, and any future code doing the hash mapping can be done using only git history, no external data would be necessary.
Moving this out of #11 as it allows for a bit more room to plan strategy regarding migrating the extensions. And might reach a more broad audience.
First of all, what would be the very best method of migrating all extensions.
All in bulk with shared git commit history
Individually with git history
Don’t care about the git history
The primary reason we should preserve git history is to not lose the credibility that previous contributors have done over the years.
Migrating extensions individually would give the benefit of having a better understanding of what’s going on, and allow fixing what’s broken more easily but it will break those commits as some commits may have touched multiple extensions. Which is not the most positive outcome for existing contributors.
The suggested approach I have in mind is:
Once that's done, I suggest we do a docbook-cs sweep so it will help contributors to make changes. We don't need to care about translators being overwhelmed by big changes.
Then finally we can prepare to publish this new doc-extensions repo by making a plan for: