Project Governance Umbrella issue

[bgrant0607]

There have been a number of discussions about project governance.

Relevant issues:

• Elders
  • Kubernetes Elders Council - proposal #28
  • initial elders page. #267
• Governance.md
  • Finalize governance.md #277
  • Update governance.md #286
  • https://docs.google.com/document/d/1UKfV4Rdqi8JcrDYOYw9epRcXY17P2FDc2MENkJjMcas/edit
• Three branches
  • Three Branches of Governance - Proposal #295
• Bylaws
  • Kubernetes Bylaws #351
• Community norms
  • Improve documentation about community norms and processes  #350

One thing that is clear is that people are trying to solve different problems. The purpose of this issue is to surface those problems and (ideally) to come to agreement about which problems we're going to tackle first. Then we get move onto proposals for how we're going to solve them.

Some problems that have been mentioned, culled from the above, in no particular order:

• The structure of the project is opaque to newcomers.
• There is no clear technical escalation path / procedure.
• There aren't consistent / official decision-making procedures for ~anything: consensus, lazy consensus, consensus-seeking, CIVS voting, etc.
• There are no official processes for adding org members, reviewers, approvers, maintainers, project leaders, etc.
• There is no official / regularly meeting body to drive overall technical vision of the project.
• We don't agree on the right level/types of engagement for leaders. Some feel that leaders should be recused from responsibilities such as SIG leadership, while others feel they need to be deeply involved in releases, etc.
• There aren't official technical leads for most subareas of the project.
• There is no centralized / authoritative means of resolving non-technical problems on the project, including staffing gaps (engineering, docs, test, release, ...), effort gaps (tragedy of the commons), expertise mismatches, priority conflicts, personnel conflicts, etc.
• In particular, there is insufficient effort on contributor experience (e.g., github tooling, project metrics), code organization and build processes, documentation, test infrastructure, and test health. Some on the project have argued that there is insufficient backpressure and/or incentives for people employed to deliver customer-facing features to spend time on issues important to overall project health.
• A related issue is counterbalancing technical- and product-based decision-making.
• Visibility across the entire project is lacking.
• Metrics, metrics, metrics, metrics. We're flying blind.
• There is no documented proposal process.
• There isn't a documented process for advancing APIs through alpha, beta, stable stages of development.
• Project technical leaders (de facto or otherwise) are not available via office hours.
• We don't have processes or documentation for onboarding new contributors.
• There are no official safeguards to prevent control over the project by a single company.
• Project leadership lacks diversity.
• There is no conflict of interest policy regarding leading/directing both the open-source project and commercialization efforts around the project.
• There is no consistent, documented process for rolling out new processes and major project changes (e.g., requiring two-factor auth, adding the approvers mechanism, moving code between repositories).
• Nobody has taken responsibility to think about and improve the structure of the project, processes, values, etc. A few people have been working on this part time, but it needs more and more consistent attention given the rate of growth of the project.
• We're also lacking people to drive, implement, communicate, and roll out improvements (and test, measure, rollback, etc.).
• There isn't a sufficiently strong feedback loop between technical contributors/leadership and the PM group.
• Nobody has taken responsibility for legal issues, license validation, trademark enforcement, etc.
• Technical conventions/principles are not sufficiently documented.
• Development practices and conventions across our repositories are not consistent.
• Our communication media are highly fragmented, which makes it hard to understand past decisions.

What other problems do people think we need to solve? Let's brainstorm first, then prioritize and cull.

cc @sarahnovotny @brendandburns @countspongebob @sebgoa @pmorie @jbeda @smarterclayton @thockin @idvoretskyi @calebamiles @philips @shtatfeld @craigmcl

[countspongebob]

Brian, great list, thanks for pulling this together.

One of the key issues for our project is whether we are philosophically running more of a top-down, hierarchically oriented project, or more of a decentralized one. I think if we can get some consensus on that (or get the BDFLs to agree), the implementation strategies become more clear.

There are good arguments both ways for how to tilt our direction.

My specific proposal was an attempt to create something more decentralized yet with enough structure to not be chaotic. Kubernetes is today effectively a BDFL model. I was attempting (perhaps poorly) to advocate for a more decentralized model.

I heard through back channels that my proposal offended a few of the BDFLs. We have awesome BDFLs, so this is not intended as criticism of any of those individuals, all of whom I have a deep personal respect for.

I believe that a different structure will lead to long term better results, yet reasonable people can disagree on this point.

If we can resolve that one critical project strategic direction, much of the rest will follow.

-Bob

[dankohn]

It's an embarrassingly small part of your list, but CNCF staff is actively working on taking "...responsibility for legal issues, license validation, trademark enforcement, etc." In particular, we'll soon be presenting a program to the K8s project leaders and the TOC of monthly retroactive license scans of all code, especially vendored libraries and incubator projects.

CNCF staff stand ready to do more (e.g., we're having discussions about spending money to improve metrics, GitHub tooling and CI).

[brendandburns]

In addition to defining roles, I'd like to see us define the process for decision making.

One other thing that concerns me with our current (lack of) governance structure is that it kind of papers over the fact that SIGs sometimes make changes that have really broad impact (sig-api-machinery, I'm looking at you...).

Currently we've been allowing SIGs to make these decisions without much input from the rest of the community. I'd like to see some notion of what size of decision is ok for a SIG to make for themselves, and what kind of decision we need to escalate to a broader group of folks. (and what that process looks like)

[bgrant0607]

One description of a few common governance structures: https://opensource.guide/leadership-and-governance/#what-are-some-of-the-common-governance-structures-for-open-source-projects

[bgrant0607]

A TBD from the governance.md PR:

• Policies/procedures for donated code (e.g., helm, kubernetes-anywhere, kompose, kargo); line is blurry between donating code to CNCF and donating code to Kubernetes; we may need to distinguish "affiliated" projects from repositories that feed into the Kubernetes releases

[sebgoa]

Incubator:

The current structure of the incubator is light weight by design but it lacks:

• Clear definition of graduation. Who votes on graduation of project ? Where are those discussions happening ?
• Processes to add members to an incubator project.

Suggestions:

• Create an Incubator Management Committee which oversees all incubator projects. Graduation is proposed to the management committee and voted by members of the committee
• Create project leadership committee which can vote on who gets promoted to committer on a project (hence be added to the author/maintainer groups).
• review/Harmoize the github group membership and document processes to maintain them.

[bgrant0607]

From a comment by @jbeda on the architectural layers doc:

• A process for accepting new features / functionality

[brendandburns]

fwiw, there data + analysis tools for the community survey data is here:

https://github.com/brendandburns/k8s-community-survey

[thockin]

When the title is "governance", it brings to mind a particular class of concerns for me. This is not to diminish the importance of the many other issues we
have as a codebase, project, and community, but I think GOVERNANCE is really the framework by which we decide who gets to decide.

1. There aren't official technical leads for subareas of the project.

2. There is no clear escalation path / procedure.

3. The org structure of the project is opaque to newcomers.

4. There is no official process for joining & progressing through org roles.

5. There is no official body to drive overall technical vision of the project.

I'm not arguing for a BDFL model, but I am wary of a fully democratic process, at least without some grounding in actual contribution/activity.

I'm not 100% sure what Bob means by "more decentralized". I'm sort of a concrete, literal person, so maybe you can get more specific with a straw man?

Rebuilding my own straw man from the other thread, from the bottom up:

---

Special Interest Groups (SIGs) are the lowest level of group ownership. The
whole project is covered by SIGs. Any topic that does not naturally fall into
a SIG must find a SIG willing to expand scope, or else petition for a new SIG
(process TBD). Any topic which naturally falls into multiple SIGs must find a
"home" SIG. SIGs can spawn limited-lifetime Working Groups (WGs) to drive
particular goals (process TBD).

Each SIG has a SIG-Lead and a SIG-TL. SIG-Leads are volunteers (process TBD)
nd need not be as deeply involved as the SIG-TL, but are responsible for
running SIG meetings, etc. SIG-TLs are drawn from the OWNERs of the relevant
area (process TBD). SIG-TLs are the deciders for technical matters within their
domains, and are responsible for technical direction and "traffic control" of
contributions. All SIG-TLs must come together on a regular basis to produce
status and planning artifacts.

The Product Management Group (PMG) is drawn from major contributing parties
(process TBD), and is responsible for driving non-technical topics, such as
cross-SIG features, deadlines, releases, etc. SIG-TLs are beholden to the PMG,
and are expected to implement the plans therefrom.

Issues that cannot be resolved satisfactorily by the SIG-TLs and PMG can be
escalated to the Elders (better name, pleeeeease), who are the final arbiters
(process TBD). The Elders are drawn from the project LEADS (process TBD), and
must convene on a regular basis to consider issues such as project scope,
architecture, overall roadmap, project health, etc.

---

There's a lot of room for refinement in there, but it gets to the heart of my
main concern - a system of roles that are clearly defined, with fairly obvious
scope, and fairly unambiguous authority. Most importantly - this is easily
memorized and is distinct from (through in most cases drawing candidates from)
the contributor ladder and org-wide roles, which deserve their own doc and
process definition.

[idvoretskyi]

The Product Management Group (PMG) is drawn from major contributing parties (process TBD), and is responsible for driving non-technical topics, such as cross-SIG features, deadlines, releases, etc.

@thockin We already have the SIG-PM [1], and its responsibility is more or less equal to that what you've described. Does your concept of PMG fit the concept of SIG-PM or PMG is rather a separate sub-organization?

1. https://github.com/kubernetes/community/tree/master/sig-pm

[thockin]

It was designed to formalize the role of SIG-PM with regards to the rest of my sketch.

…

On Thu, Mar 16, 2017 at 3:22 AM, Ihor Dvoretskyi ***@***.***> wrote: The Product Management Group (PMG) is drawn from major contributing parties (process TBD), and is responsible for driving non-technical topics, such as cross-SIG features, deadlines, releases, etc. We already have the SIG-PM [1], and its responsibility is more or less equal to that what you've described. Does your concept of PMG fit the concept of SIG-PM or PMG is rather a separate sub-organization? 1. https://github.com/kubernetes/community/tree/master/sig-pm — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#402 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVGwhnZW5jfykOjOlWH9tw6SdXZu8ks5rmQ1ugaJpZM4MLmBs> .

[jbeda]

@thockin I like your sketch.

But I do worry about the phrasing: "SIG-TLs are beholden to the PMG,
and are expected to implement the plans therefrom." This implies a level of command and control that may not be practical. In an OSS project we can't really tell anyone where to spend their time. We can withhold support, not merge, not review, etc. But we can't make stuff happen if folks aren't bought in.

Also, while we have a clear (ish) idea of what it means to be a technical contributor (PRs, reviews, etc). We don't have an idea for the product side. We can either require that folks that have deciding power in the PMG must also be contributors or we have to come up with some other way for folks in that group to prove themselves.

Concretely, I don't want a big company to one (or many) people who call themselves PMs and have them come in and try to direct the project without establishing themselves in the community. Part of the ethos of our community is that the project must come ahead of commercial concerns. That can only be shown through actions over time.

[thockin]

On Thu, Mar 16, 2017 at 9:59 AM, Joe Beda ***@***.***> wrote: @thockin I like your sketch. But I do worry about the phrasing: "SIG-TLs are beholden to the PMG, and are expected to implement the plans therefrom." This implies a level of command and control that may not be practical. In an OSS project we can't really tell anyone where to spend their time. We can withhold support, not merge, not review, etc. But we can't make stuff happen if folks aren't bought in.

Sure, I thought about that as I was writing, but I wasn't sure the right words. Like any constitution, we need to find the right balance of powers and the right checks on those powers.

Also, while we have a clear (ish) idea of what it means to be a technical contributor (PRs, reviews, etc). We don't have an idea for the product side. We can either require that folks that have deciding power in the PMG must also be contributors or we have to come up with some other way for folks in that group to prove themselves.

Agree. What if PMG is elected by the OWNERS or MAINTAINERS to represent the product interests. In a way, it's still an engineering driven project, but with a refinement from actual product people. E.g. if there are enough MAINTAINERS from company X, they can get a PM on the PMG; but they could also just demonstrate enough PM-contribution to be elected naturally. There can be PM contributors that are not elected to PMG just as there are technical contributors who are not MAINTAINERs. I am sure there's some vote distribution technique that prevents non-obvious gaming...

Concretely, I don't want a big company to one (or many) people who call themselves PMs and have them come in and try to direct the project without establishing themselves in the community. Part of the ethos of our community is that the project must come ahead of commercial concerns. That can only be shown through actions over time.

100% agreed.

[countspongebob]

@thockin : Three branches proposal is my concrete proposal for distributing project control, and concretely, this proposal gives the PM SIG too much weight without a check.

Inherent in my proposals is an opinion that the the project is currently ad-hoc highly centralized, and should be formally more decentralized. This means more people from more companies granted a higher level of trust and empowerment. I grant that this will feels like a risk, but the reward will be a higher project velocity and more adaptability to future requirements.

This is the major lesson of micro-service architectures and high velocity organizations and we should move in this direction, not away from it.

Presently the PM group (as product management groups often are) is release-based and feature oriented. We must have a strong orientation to architectural improvement over the long haul. This group currently does not have the participation from the technical community. We have seen in recent years large open source projects go down the wrong path on this front.

This, as Joe points out, is the point where the project is the most susceptible to disruption from those without technical credentials and history on the project, and we need to protect the project from that with appropriate checks.

Project policy and process issues very much need to be separated from this group. Process efforts centered on product management teams generally err in one direction, process efforts run by developers alone generally err in the other. We need to find a place where some balance can be achieved.

If I were to extend the separation-of-powers concept to the SIGs, I would propose that every SIG have:

• A rep from the PM group who attends all meetings and represents the conduit of feature-related information in both directions. Not control - communication and leadership.
• An organizer (the current model for the SIG lead) who ensures process items for the meetings are followed, especially the transparency of the proceedings, and represents process issues back to the policy board.
• Technical lead(s) who have the requisite project credentials and permissions and represent the council of elders.

I believe conflating these roles will counter-productive.

A related lesson is that without debate is that the the high velocity organization MUST have great automated testing. Formalizing command-and-control organization is a very typical response to testing automation that has too many gaps.

Upon reflection of this point, this would indicate that the testing SIG should probably have some additional empowerment related to setting acceptable bars for releases.

In summary: We need to find the way towards more de-centralizations, transparency, and empowerment and away from centralized control hierarchies.

• PM group should be oriented towards roadmaps, communication, and leadership, not management and control.
• SIG testing needs greater empowerment to gate releases
• Architecture and technical standard setting need more formalization (council of elders).
• More engineers on the project need more empowerment.

-Bob

[thockin]

On Fri, Mar 17, 2017 at 9:23 AM, Bob Wise ***@***.***> wrote: @thockin : Three branches proposal is my concrete proposal for distributing project control, and concretely, this proposal gives the PM SIG too much weight without a check.

To be sure, my proposal here was somewhat off the cuff, especially with regards to PMG. I'm going to pause here and re-digest your proposal. OK, I am back. Bob, I think we're about 84.7113% aligned. I de-emphasized what you call the Project Policy Board, probably because I am not sure I understand it, and I added the SIG-TLs. More on this below, after I talk to your whole message.

Inherent in my proposals is an opinion that the the project is currently ad-hoc highly centralized, and should be formally more decentralized. This means more people from more companies granted a higher level of trust and empowerment. I grant that this will feels like a risk, but the reward will be a higher project velocity and more adaptability to future requirements.

I'm not getting that from your proposal. You're endorsing "PMG" for for features / roadmap / product issues, as am I. You're endorsing "elders", as am I. You keep describing this as "more decentralized", but where is that effect in your proposal? Is the main thrust that you're divvying up what PMG currently does into 2 disjoint groups? I'm probably OK with that, but I do have a human concern - when you give a group of people an unbounded mandate and measure their "worth" by it, they are going to do things. Creating a group whose focus is "creating policies", we're going to get policies. How many policies do we want? We certainly need some, probably more than we have today, but does that warrant a full branch of governance?

This is the major lesson of micro-service architectures and high velocity organizations and we should move in this direction, not away from it.

The associated lesson is that microservice architectures encourage siloing, focusing on local optima, and general navel-gazing.

Presently the PM group (as product management groups often are) is release-based and feature oriented. We must have a strong orientation to architectural improvement over the long haul. This group currently does not have the participation from the technical community. We have seen in recent years large open source projects go down the wrong path on this front.

OK, ACK. We have been very product/feature focused as a project (rightly so, I think, but that eventually has to stabilize). What part of your proposal fixes this, though?

This, as Joe points out, is the point where the project is the most susceptible to disruption from those without technical credentials and history on the project, and we need to protect the project from that with appropriate checks. Project policy and process issues very much need to be separated from this group. Process efforts centered on product management teams generally err in one direction, process efforts run by developers alone generally err in the other. We need to find a place where some balance can be achieved.

Doesn't this suggest that the process-related group is a "reconciliation group" between PMG and eng (what I called the SIG TLs), rather than a standing body?

If I were to extend the separation-of-powers concept to the SIGs, I would propose that every SIG have: A rep from the PM group who attends all meetings and represents the conduit of feature-related information in both directions. Not control - communication and leadership. An organizer (the current model for the SIG lead) who ensures process items for the meetings are followed, especially the transparency of the proceedings, and represents process issues back to the policy board. Technical lead(s) who have the requisite project credentials and permissions and represent the council of elders.

Excellent, our alignment just went up. Can you clarify what "represent the council of elders" means in your mind? Is that a top-down model, where the N<=5 elders dictate where the project should go, and N<=20 SIG-TLs execute? Or is that a decentralized model where SIG-TLs decide where the project should go, and the elders adjudicate and approve?

I believe conflating these roles will counter-productive.

Sorry, I just want to clarify - conflating SIG-Lead, SIG-TL, and SIG-PM will be counter-productive? I agree.

A related lesson is that without debate is that the the high velocity organization MUST have great automated testing. Formalizing command-and-control organization is a very typical response to testing automation that has too many gaps. Upon reflection of this point, this would indicate that the testing SIG should probably have some additional empowerment related to setting acceptable bars for releases.

I'm OK with that in broad strokes.

In summary: We need to find the way towards more de-centralizations, transparency, and empowerment and away from centralized control hierarchies. PM group should be oriented towards roadmaps, communication, and leadership, not management and control. SIG testing needs greater empowerment to gate releases Architecture and technical standard setting need more formalization (council of elders). More engineers on the project need more empowerment.

So after reading this, I really do think we (you ad I, everyone else can catch up) are pretty close. I would describe it as: * Each SIG has a lead (administrator of the group), a TL, and a PM. SIGs can spawn limited-duration working groups. * SIG-TLs (hereafter just "TLs") are the primary technical owners of subsystems, and can delegate ownership as appropriate, with the SIG-Lead. * Elders (wow, I hate that name) own broad architectural concerns and cross-SIG issues, provide advice and consent on significant SIG-driven issues, and adjudicate all escalations. * PMG drives product-wide feature goals, roadmaps, and releases. * A joint group of TLs and PMs forms the policy group, which drives project-wide processes, as needed. * Certain additional responsibilities and powers are granted to the testing and docs SIGs wrt releases and policy definition * The contributor ladder defines roles and the criteria for "promotion". * Process for who is eligible for each group, how they are chosen, and how they are removed is TBD. Am I getting closer to what you're thinking about? If not, where am I missing?