Pattern Idea : Using Extensions to manage Contributions at scale

[Ssukriti]

Might be related to https://github.com/InnerSourceCommons/InnerSourcePatterns/blob/main/patterns/1-initial/reluctance-to-accept-contributions.md

Problem
 As the number of new feature contributions to a mature inner-source repository rapidly increase, it adds more burden on code reviews and maintenance. How do we allow for faster release of new features, encouraging innovation and experimentation; while also keeping repositories well maintained?
 
 Context:

• More organizations or company employees are contributing to strategic inner-source codebases, some of which are scaling rapidly with new feature contributions. This is adding a huge code review backlog on a smaller group of maintainers, resulting in maintainer burnout as well as slowing down release of new capabilities.

• Adding an excessive number of capabilities and code to repositories without due diligence is making them difficult to maintain. 

• New features are getting released without thorough documentation, hardening or testing, creating a poor user experience.

• You want to allow for expansion and encourage innovation and experimentation without restricting contributions, while also keeping good code and quality standards for user experience. 

• A large amount of time goes into hardening and thorough testing of features to meet product standards, but you want to allow for faster release of new innovations for adopting products to explore before investing time in productionization of capabilities.

• You want to encourage the community to share innovations, combining product capabilities with other use-cases without adding more dependencies to primary repository.

Solutions
Allowing extensions/plugins to high-scale inner-source codebases or libraries can relieve the maintenance burden on repository maintainers and allow faster release of new features for adopting products to explore. This shifts maintenance of capabilities to extension owners and allows the primary repository to support capabilities that have been adopted more widely and are more strategic.

• Create library or repository extension templates to allow addition of new features in new repositories, retaining the same modular structure and APIs as the primary repository. 

• Loosen the requirements for contributors to create extensions by bypassing reviews to allow for faster release or experimentation.

• Allow registration of extensions with primary libraries for users to leverage extensions alongside the original project.

• Make extensions easily discoverable with a publishing page showing the best extensions that users have created, which are ready for product usage.

• Establish the lifecycle for extensions from creation to porting into the primary library. For instance:

1. Extension creators continue maintaining the extension, providing any support and fixing defects. Any extension left unmaintained will be unlisted from the publishing page.
2. Create criteria for when an extension can be ported to the primary repository, such as adoption of the extension by internal products and demand for the feature. 
3. Porting process of the extension to the primary repository will follow more stringent code review guidelines as set by library maintainers.

Following the principles outlined above ensures that:

• Contributing an extension feels identical to adding a new feature to the primary repository and is a similar amount of work.

• Extensions are discoverable in a repeatable manner to all users of the primary project; just because code doesn't live in the main repository yet does not mean it is not valuable.

• The maintainer burden is reduced until an extension has demonstrated that it fills an important gap in the primary project.

Authors:
Sukriti Sharma
Alexander Brooks

[Ssukriti]

I am unable to attach labels, but this is an initial idea that I will contribute if you all think is worthwhile to have as a pattern.
We have adopted this in our organization and have seen success at managing inner-source projects at scale

[NewMexicoKid]

Sounds like an interesting idea worth documenting as a pattern. The last three points in the context sound more like forces (highlighting trade-offs). I wonder if there is something more to the solution (e.g., architecturally, are there ways to design the framework for extensions or plugins so that there is sufficiently loose coupling and enough future-proofing to enable the base software to grow and extend without requiring dependent changes in the extensions; and is there a requirement for automated tests to ensure that the base software doesn't break extensions or that new extensions don't negatively impact performance of the software, etc).

…

--Tim

On Mon, Aug 15, 2022 at 7:27 PM Ssukriti ***@***.***> wrote: I am unable to attach labels, but this is an initial idea that I will contribute if you all think is worthwhile to have as a pattern. We have adopted this in our organization and have seen success at managing inner-source projects at scale — Reply to this email directly, view it on GitHub <#439 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARUJ4FEKEVDMWPIUR7H5LDVZLOAJANCNFSM56T44NUA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

-- Tim Yao - ***@***.*** ***@***.***> ***@***.***> aka NewMexicoKid at National Novel Writing Month, http://nanowrimo.org <http://www.nanowrimo.org> co-Municipal Liaison for the Illinois::Naperville region NaperWriMo - http://naperwrimo.org

[Ssukriti]

Thank you for the feedback @NewMexicoKid ! Yes you are right, I will move the last three points to forces as tradeoffs while drafting the pull request.

Some thoughts on the architecture:

1. To test that the base software doesn't break extensions:
   Extension templates will require the base software to be installed a dependency and they will build on top of that. Since the extension will be in a separate Github repository, we can also include the same performance testing framework in the extension repositories, where the tests can run in an automatic fashion with every version upgrade of base software. This way the ownership and maintenance of the tests with extensions would be on the extension owners. If any changes in the base software cause the extension to break, the owner can either limit the compatibility to older versions or will have to update the extension accordingly to adopt latest versions of base software. (we think this scenario should be rare unless there is a major release upgrade of base software)

We would avoid adding more tests to the base repository in order to keep it lightweight and have lesser maintenance on primary maintainer group.

2. tests to validate that new extensions don't negatively impact performance of the software:
   This is an interesting point and maybe a way to include this would be that the extensions repositories include all the performance tests for all the capabilities offered by the base software as well . Any drop in performance should be documented in the extension so users are made aware of the downsides.

About future-proofing and long term maintenance, I think we could use more brainstorming, but in general changes required on extensions side to keep up with base repository could be minimized with a modular structure in base software, such that unless there are major API changes (major release), the functionality remains independent. We also allow existing feature enhancements directly into the primary repository and recommend extensions only for a new use case or capability.

Having a well defined cycle for accepting the extension into primary library could also help move the strategic capabilities into the primary repository sooner. (such as if the extension received adoption by X products. ) If an extension was not able to build a community around it, such that maintenance had to dropped after X amount of time, it would be unlisted, which is also fine I think as it would be similar to deprecating features in software. We need to do our best to offer the same discoverability to extension modules to give them a fair chance to be adopted and build a community/ leverage the community of the base repository.

[spier]

@Ssukriti thank you for sharing this approach with us!

I especially like that it addresses challenges related to "software architecture for InnerSource".
For those we don't have many patterns yet.

For inspiration you could look at these patterns.
They address quite the same thing as your Extensions idea though.

• pattern draft: Incubator Pipeline #338
• https://github.com/InnerSourceCommons/InnerSourcePatterns/blob/main/patterns/1-initial/modular-code.md

You could theoretically submit your pattern as Structured immediately (Level 2), as you have one Known Instance already (your own org).

However I would recommend to start by submitting it as Initial (Level 1), and then maturing it over time to Level 2.
That has these benefits:

• merging level 1 patterns into our repo is a faster process
• you can collect more feedback, as your pattern is shared in our repo and can get more eyeballs on it
• the pattern won't be included in our book immediately (which e.g. means that it is still easier to rename the pattern, etc)

If this approach works for you, you would start this by:

• forking this repo
• creating a new branch for your pattern
• adding a new pattern file to the patterns/1-initial folder
• entering your pattern content into that file, by copying the Pattern Template and filling it in
• then submitting your new pattern as a PR

Also take a look at our Contributor Handbook. The requirement descriptions or Level 1 patterns should be helpful.

I hope that you will find the contribution experience to be a smooth one.
We are here to help and also open to any feedback for improving our process.

[NewMexicoKid]

I think that these answers you've provided for my questions about architecture and maintenance are great points to raise in the solution; and it seems like they address some additional forces/trade-offs that could be listed. It would probably be good to also note what the Resulting Context is after the Solution is applied. Sometimes the application of the Solution introduces new forces and problems that need to be addressed in subsequent patterns (e.g., what happens when the extension framework doesn't go as far as an extension writer needs it to go?). Looking forward to reviewing the pattern. Thanks! --Tim

…

On Tue, Aug 16, 2022 at 1:15 AM Sukriti Sharma ***@***.***> wrote: Thank you for the feedback @NewMexicoKid <https://github.com/NewMexicoKid> ! Yes you are right, I will move the last three points to forces as tradeoffs while drafting the pull request. Some thoughts on the architecture: 1. To test that the base software doesn't break extensions: Extension templates will require the base software to be installed a dependency and they will build on top of that. Since the extension will be in a separate Github repository, we can also include the same performance testing framework in the extension repositories, where the tests can run in an automatic fashion with every version upgrade of base software. This way the ownership and maintenance of the tests with extensions would be on the extension owners. If any changes in the base software cause the extension to break, the owner can either limit the compatibility to older versions or will have to update the extension accordingly to adopt latest versions of base software. (*we think this scenario should be rare unless there is a major release upgrade of base software*) We would avoid adding more tests to the base repository in order to keep it lightweight and have lesser maintenance on primary maintainer group. 1. tests to validate that new extensions don't negatively impact performance of the software: This is an interesting point and maybe a way to include this would be that the extensions repositories include all the performance tests for all the capabilities offered by the base software as well . Any drop in performance should be documented in the extension so users are made aware of the downsides. About future-proofing and long term maintenance, I think we could use more brainstorming, but in general changes required on extensions side to keep up with base repository could be minimized with a modular structure, such that unless there are major API changes (major release), the functionality remains independent. We also allow existing feature enhancements directly into the primary repository and recommend extensions only for a new use case or capability. Having a well defined cycle for accepting the extension into primary library could also help move the strategic capabilities into the primary repository sooner. (such as if the extension received adoption by X products. ) If an extension was not able to build a community around it, such that maintenance had to dropped after X amount of time, it would be unlisted, which is also fine I think as it would be similar to deprecating features in software. We need to do our best to offer the same discoverability to extension modules to give them a fair chance to be adopted and build a community. — Reply to this email directly, view it on GitHub <#439 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARUJ4DM2DQMKBE7Z6JXB7DVZMWWDANCNFSM56T44NUA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Tim Yao - ***@***.*** ***@***.***> ***@***.***> aka NewMexicoKid at National Novel Writing Month, http://nanowrimo.org <http://www.nanowrimo.org> co-Municipal Liaison for the Illinois::Naperville region NaperWriMo - http://naperwrimo.org

[Ssukriti]

I have opened a pull request with the submission as an initial idea #444

Thanks

[spier]

@Ssukriti as promised the pattern is now merged and available here:
https://github.com/InnerSourceCommons/InnerSourcePatterns/blob/main/patterns/1-initial/extensions-to-manage-contributions-at-scale.md

I am closing this issue here as well.

Let's open new issues and PRs as we see fit to improve this pattern further in the future.

And thanks again for sharing your great pattern!