Update DEVELOPER_GUIDE.md to document bwc and feature flag mechanims

[nknize]

OpenSearch uses @opensearch.internal, @opensearch.api, @opensearch.experimental, and feature flags to signal maturity and usage intent of the various implementation components. Each one of these annotations carry a level of backwards compatibility guarantees and support requirements that is not officially documented. This change officially documents those bwc requirements, level of guarantee, and recommended approaches in how to best use the mechanisms to communicate this to the development and user community.

relates #2868

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock

• URL: https://build.ci.opensearch.org/job/gradle-check/9974/
• CommitID: 8c72bd8
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWhenAllNodesExceededHighWatermark

• URL: https://build.ci.opensearch.org/job/gradle-check/9976/
• CommitID: 90b8e4e
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[dblock]

We have opensearch-project/.github#108 opened on the topic. Care to take a look? How much does this paragraph resolve that? Should to go to a higher level doc so it can apply to all software in opensearch-project?

[nknize]

I think we can keep that open to discuss broader (project wide) adoption? I think there are some top level questions for the project as a whole (e.g., do we want to force implementation details or just standardize around a process?) that we can continue discussing while documenting here how it's done concretely in this OpenSearch repo.

[dblock]

Sounds good. I just wanted to highlight that issue for awareness. We don't want for a single repo to contradict a top level doc, and you rightfully call out that we want to talk only process, not implementation, at the org level.

[dblock]

I like the text here, it's a step forward. Only concerns is how it applies to the entire distribution vs. core.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testStartReplicaAfterPrimaryIndexesDocs

• URL: https://build.ci.opensearch.org/job/gradle-check/9979/
• CommitID: 74d8d69
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[reta]

The wording and intent make sense to me, I have a general question on @opensearch.api. The thing we do not enforce (but probably should) is that @opensearch.api should expose only other APIs that are marked as @opensearch.api (obviously, internally could use anything as long as it does not leak outside). I believe we have a gap here in a way that API which is marked as stable does (or may) expose the components which are not stable (@opensearch.internal or no annotations at all). Does it make sense?

[andrross]

I like the text here too.

Only concerns is how it applies to the entire distribution vs. core.

The idea I'm coming around to, and has been hinted at by some of the other discussions on this topic, is that strict semantic versioning is the right approach for "Data", "User API", and the new plugin SDK under development, but maybe we should be a little less strict for the tightly-coupled "legacy" plugin APIs. As @reta pointed out above even applying semantic versioning to only the things labeled @opensearch.api will be challenging as the API leaks lots of internal types. A policy allowing for the deprecate-then-remove cycle within a major version for these APIs would free up the evolution of these code bases without also forcing a major version update on all users. Is this worth a broader discussion?

[nknize]

I agree with all of these great points! Internal Leakage (I'm declaring that an official word - thx @andrross 😄) whether intentional or not, is absolutely a problem w/ today's code organization. I'm trying to move the needle here by incrementally formalizing and defining some of these things w/ you all so we can asynchronously take on the refactor effort. As @dblock pointed out, it's a big undertaking for one person or one project. By formally defining these mechanisms and purpose for each library and module, I think we'll lift enough of the "fog of war" to divide an conquer as a community w/o turning this into a corporate effort

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-6028-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 67e3b1972c0222528ce48e1e303b7d34c9f4906e
# Push it to GitHub
git push --set-upstream origin backport/backport-6028-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-6028-to-2.x.

[andrross]

Backport here: #6047