mdBook for multi-repo github org

[fmuyassarov]

Hi,
Thanks a lot first of all for the amazing work with mdBook.

I wanted to discuss/check out with mdBook contributors if there is a way to solve the problem we are facing currently.

Basically, we have multiple repositories under a single github organization. Each repository has its own documentation. What we want is that - a single mdBook based book that would fetch documentations from all our repositories and show it to the user as a single book.

I will try to use example repositories to be more clear. Let’s say we have example github organization, with x, y, z repositories under it.

example
 ├── x
 ├── y
 ├── z

Our wish is - whenever there is a pull request in any of the x, y, z repositories that introduces documentation changes, we want to have a preview with netlify on the corresponding repository. This brings the following problems:

1. If we want to have netlify preview on each PR - that means we need to have all the html, css, etc files (generated by mdbook) for each x, y, z repository
2. Whenever, we do the preview for a single repo, we don’t have anymore the global view of the book, it will be narrowed down only to a single repo part

One way that we came up was that - we have a separate repo w for storing documentation of all the other repositories. But we are not moving the docs from x, y, x to w but instead we clone x, y, and z as submodules into the w repo. The reason for not moving them to the w repo is because we would like to keep the documentation as close as possible to the repository that it corresponds to. This will work, we could start the mdbook from w repo + including content of x, y, z that we submoduled. And there will be automated CI that will be updating w repo for example every night with the changes introduced in x, y, z. But this also brings some problems when renewing a PR

1. Example: there is a PR on x repo and we want to show the changes on netlify to the PR author. So, netlify will try to fetch the content from the w repository(which doesn’t include current commit in x repo). But since w repo has no idea about the PR on x repo, we can’t really show the preview with the actual changes being introduced with the current commit.

In short it is problematic to have a single book that will be sourcing contents from multiple repositories.

I would really love to hear your inputs and open for a discussion.
Thanks in advance.

[fmuyassarov]

/cc @Michael-F-Bryan,
/cc @budziq,
/cc @steveklabnik,
/cc @frewsxcv

[frewsxcv]

Example: there is a PR on x repo and we want to show the changes on netlify to the PR author. So, netlify will try to fetch the content from the w repository(which doesn’t include current commit in x repo). But since w repo has no idea about the PR on x repo, we can’t really show the preview with the actual changes being introduced with the current commit.

Is there something stopping you from updating the x git submodule in the w repository in netlify?

[fmuyassarov]

Example: there is a PR on x repo and we want to show the changes on netlify to the PR author. So, netlify will try to fetch the content from the w repository(which doesn’t include current commit in x repo). But since w repo has no idea about the PR on x repo, we can’t really show the preview with the actual changes being introduced with the current commit.

Is there something stopping you from updating the x git submodule in the w repository in netlify?

No, there isn't. I actually didn't know that we could do it from within the Netlify.
Is the following scenario what you have in your mind?

1. PR is opened in x repo
2. CI triggers Netlify deploy
3. Netlify updates x git submodule in w repository and deploys it from w repository?

[frewsxcv]

Admittedly I'm not familiar with Netlify, but I'd hope there's a way to execute git commands. In any case, is there a feature request here for mdBook? Wondering what action there is for mdBook maintainers to take from your workflow you're attempting

[fmuyassarov]

I’m not sure if I can call it a feature request. My intention was to discuss with the mdBook community to see if I can get some help on my specific issue. Because, we are interested in using mdBook and I thought maybe you have experienced a similar challenge.

[Michael-F-Bryan]

What about giving SUMMARY.md a way to #include other SUMMARY.md files?

That way you could have a top level SUMMARY.md file like this:

# Summary

- [overview](index.md)
- {{#import "Repository A" "./a/book/src/SUMMARY.md"}}
- {{#import "Repository B" "./b/book/src/SUMMARY.md"}}
- [Conclusion](conclusion.md)

And SUMMARY.md in your a submodule:

- [First](first.md)
- [Second](second.md)

And SUMMARY.md in your b submodule:

- [First](first.md)
- [Second](second.md)
- [Third](third.md)

(the duplicate names are intentional)

Then the parser would expand all annotations of the form {{#import <title> <path_to_summary_md> }} to get something like this:

# Summary

- [overview](index.md)
- Repoistory A
  - [First](./a/book/src/first.md)
  - [Second](./a/book/src/second.md)
- Repository B
  - [First](./b/book/src/first.md)
  - [Second](./b/book/src/second.md)
  - [Third](./b/book/src/third.md)
- [Conclusion](conclusion.md)

Then whenever you make docs changes in a submodule the top level repository does a git submodule update --remote and regenerates the docs, and we automatically pick up all the new pages.

I'm not sure how this import mechanism would interact with things set in the submodule's book.toml. Maybe when including other books as a subtree we would only grab the content, but that means we don't do any preprocessing work or include additional JS or CSS.

[fmuyassarov]

@Michael-F-Bryan Thanks for the comment.

I really like the idea of #include. Of course it doesn't solve the whole problem, but it would simplify our life because we won't need to do manual updates on the top level SUMMARY.md in case of some changes in one of the other SUMMARY.md files.

And git submodule update .. should definitely work to keep w repository up to date with docs changes introduces in any other submodules.

[yozhgoor]

I have an issue like this, i use mdbook as an Onenote like for now. I have one book and i write multiple book in it. It's ok for 2-3 book but for more is not easily readable.

What about and architecture like this:

example
├── x
├── y
├── z

but with this:

 main directory
 ├── book 1
 ├── book 2
 ├── book 3

I don't think its gonna solve the whole problem about Netlify but, for my use case it would be a must.
Note: I'm a newbie so, i try. If I'm missing something or if a way to solve this problem, don't hesitate to tell me

[max-sixty]

In case anyone finds this helpful:

We made an extremely hacky approach work for this sort of thing. While we don't have multiple repos, we want to compile code as part of the book building process. So we abuse the preprocessor hooks to do that, both for a compiled text editor, and an in-tree preprocessor: https://github.com/prql/prql/blob/ae053d9c5547f91317ea11cf5f957270ca366804/reference/book.toml

[liufuyang]

This feature could be something nice to have. Plus on this, it might be even helpful if we could allow mdbook to print all the markdown files under a specific folder recursively and the pages just organized by path and file names, or the first line of # xxx as title? (perhaps related to #1016)

If this is can be done, then in repo A we can have

docs
 |--foder1
 |     |-- A1.md
 |--Readme.md  

and in repo B we can have

docs
 |--foder1
 |     |-- B1.md
 |--Readme.md  

Then in the summary of another "main" repo, where we have folder a,b as submodules
and a summary file can be like:

- [Header](index.md)
- Repository A(recursive{./a/docs}) # auto recursive build all the md files under that path
- Repository B(recursive{./b/docs}) # auto recursive build all the md files under that path
- [Footer](footer.md)

This could be very helpful for small orgnizations to use mdbook to create a centralized doc site for multiple of their github repositories. And mdbook config is only in the main/master repo, while all the md fils are just laying around in each repo. Each repo owner can use file structure and first line of the md file to define the chapter/page headlines.

[cbrake]

I also use mdbook with multiple repos as it's most effective to keep docs close to the source. The SUMMARY.md at least makes this possible. One minor problem is the edit link does not work in submodules. Looking forward to additional improvements!