Add design process section to the docs

[alamb]

Which issue does this PR close?

• part of [Epic] Better / Improved Documentation, Tutorials and Examples #7013

Rationale for this change

While discussing a design for cancellation with @pepijnve and @zhuqi-lucas and myself, @ozankabak wrote a great summary of how the DataFusion community handles larger projects:

• feat: Allow cancelling of grouping operations which are CPU bound #16196 (comment)

Look, I see that you are trying to help and we do want to take it. I suspect we might be facing a "culture" challenge here: Typically, DF community attacks large problems by solving them bit by bit and refining a solution iteratively. This is unlike some other projects which front-load the effort by going through a more comprehensive design process. We also do that for some tasks where this iterative approach is not applicable, but it is not very common.

This "bit by bit approach" doesn't always succeed, every now and then it happens that we get stuck or go down the wrong path for a while, and then change tacks. However, we still typically prefer to "advance the front" and make progress in tangible ways as much as we can (if we see a way). This necessarily results in imperfect solutions being the "state of the code" in some cases, and they survive in the codebase for a while, but we are good at driving things to completion in the long run.

I really liked that description and think it captures well the current state of the project, and thus would be valuable to make part of the docs

What changes are included in this PR?

Add a description of the design process to the Datafusion docs site

Are these changes tested?

By CI

Are there any user-facing changes?

New docs

[zhuqi-lucas]

LGTM Thank you @alamb!

[ozankabak]

Looks good, thanks @alamb

[alamb]

Of course we always have to be 🎣

[jonathanc-n]

This is really nice, thanks @alamb!

[alamb]

This is really nice, thanks @alamb!

Thanks -- I was just channeling @ozankabak :)

[pepijnve]

Sorry to go a bit off topic for a sec, but there's some context I would like to add. I worked on API design of a commercial software library with tons of extension points for 10+ years where backwards compatibility of the public API was something we stuck to religiously because of the burden API breakage places on the entire user base. Doing that kind of work for an extended period of time makes you think three times about new API and all the hypothetical uses and abuses; perhaps a bit too much.

Relevant question to this text and the project is what the project's stance is wrt API stability? Merging fast means you're likely to ship something a little bit too quickly every now and then. I'm not saying it's a bad strategy, just wondering how you balance the tension between stability and velocity.

[alamb]

Relevant question to this text and the project is what the project's stance is wrt API stability? Merging fast means you're likely to ship something a little bit too quickly every now and then. I'm not saying it's a bad strategy, just wondering how you balance the tension between stability and velocity.

I would say we "try not to do API churn but it happens every release". Indeed it has come up as a challenge for downstream users, though I would say it has been less of a challenge the last 6 months or so. There is more detail here

• [DISCUSSION] Making it easier to use DataFusion (lessons from GlareDB) #13525

The policy is documented here: https://datafusion.apache.org/contributor-guide/api-health.html

You can get a sense of the kinds of changes required by looking at https://datafusion.apache.org/library-user-guide/upgrading.html

Basically at some point I expect users of DataFusion will care enough about non breaking releases that they will want to contribute to helping make some release vehicle that has stable APIs (e.g. backport stuff to a LTS release for example)

But until that happens we just keep cranking on the code and change APIs every month

[comphead]

Suggested change

	However, this "bit by bit approach" doesn't always succeed, and sometimes we get
	However, this "bit by bit approach" doesn't always succeed, and sometimes we get

wondering if this is needed?

[alamb]

I think the idea behind this sentence is to acknowledge the tradeoffs inherent in "design / build" vs "big design all upfront" (it is this tension that actually sparked the original comment in the first place))