[PROPOSAL]Merge Generated APIs into Existing Client Module

[saimedhi]

What are you proposing?

Merge the generated APIs into the existing client module to progress towards complete client generation, ensuring alignment with the server.

How does the opensearch-py API Generator function?

1. Run the generator as per the developer guide.
2. The generator retrieves the OpenSearch API specification JSON data.
3. API paths and methods are processed and grouped based on the x-operation-group field.
4. API parameters, parts, methods, and documentation are updated and organized.
5. For each API endpoint, instances of Module and API classes are initialized.
6. Generator checks if corresponding module exists in the clients folder for each API's namespace.
7. If the module is absent, creates a module for the corresponding namespace.
8. In the module, a class is created based on the namespace, containing API definitions or actions.
9. Ninja templates are utilized for generation, with overrides for compatibility.
10. Format the code using instructions in developer guide.
11. If a namespace/module already exists, then existing definitions in that module are replaced by the generated API definitions.

What problems are you trying to solve?

Consider 'indices' namespace, post-generation, existing APIs are replaced by generated ones. Some generated API may be different from the existing APIs. We need to explore solutions for handling divergent APIs, termed as 'Patch APIs.' Our goal is to structure the merging process to prevent breaking changes. We categorize APIs as 'generatable' or 'Patch'

Solutions:

Three approaches are discussed below to separate and structure generatable and patch APIs.

Current client structure:

Approach 1

Consider "indices" namespace, all the patch APIs are placed in a separate file "indices/patch.py". In this way while running generator, patch APIs wont be altered. And we generate all other APIs into "indices.py". Easy implementation and contributor-friendly.

Approach 2

Change client structure to one file per action/API within the namespace. Significant effort initially, both in changing the client structure and adapting the generator. The following PR shows more about this approach. https://github.com/opensearch-project/opensearch-py/pull/475/files

Approach 3

Modify specs of patch APIs within the generator to match existing APIs. Utilize overrides in ninja templates to overwrite generated APIs. Time-consuming but avoids major structural changes.

What is the developer experience going to be?

Developers must understand generator functionality before contributing. Decisions on contributions (API spec, generator, or patch) are essential.

Are there breaking changes to the User Experience?

Aims for no breaking changes; user experience remains consistent.

Why should it be built?

Facilitates easier addition of client support for new server-side APIs.

Please let me know your thoughts and suggestions.

[dblock]

This is quite thoughtful. If you didn't have to minimize breaking changes, what would this process look like? I would work backwards from there, with the hope that once the API generator is producing the entire API we can delete all patches and make one major version bump breaking backwards compatibility.