Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: generate a GAPIC library from api definition #3208

Merged
merged 23 commits into from
Sep 20, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
change to api_definitions_path
  • Loading branch information
JoeWang1127 committed Sep 20, 2024
commit 5817ea924bdca4e92a1d57b2de4eaac7780939e3
6 changes: 3 additions & 3 deletions .github/scripts/hermetic_library_generation.sh
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ git show "${target_branch}":"${generation_config}" > "${baseline_generation_conf
# get .m2 folder so it's mapped into the docker container
m2_folder=$(dirname "$(mvn help:evaluate -Dexpression=settings.localRepository -q -DforceStdout)")

# download api definition from googleapis repository
# download api definitions from googleapis repository
googleapis_commitish=$(grep googleapis_commitish "${generation_config}" | cut -d ":" -f 2 | xargs)
api_def_dir=$(mktemp -d)
git clone https://github.com/googleapis/googleapis.git "${api_def_dir}"
Expand All @@ -103,9 +103,9 @@ docker run \
gcr.io/cloud-devrel-public-resources/java-library-generation:"${image_tag}" \
--baseline-generation-config-path="${workspace_name}/${baseline_generation_config}" \
--current-generation-config-path="${workspace_name}/${generation_config}" \
--api-definition-path="${workspace_name}/googleapis"
--api-definitions-path="${workspace_name}/googleapis"

# remove api definition after generation
# remove api definitions after generation
rm -rf "${api_def_dir}"

# commit the change to the pull request.
Expand Down
8 changes: 4 additions & 4 deletions library_generation/DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ generate a GAPIC repository with a given api definition (proto, service yaml).
For example, googleapis
```
git clone https://github.com/googleapis/googleapis
export api_definition_path="$(pwd)/googleapis"
export api_definitions_path="$(pwd)/googleapis"
```

### Download the repo
Expand All @@ -126,7 +126,7 @@ python -m pip install .
```
python cli/entry_point.py generate \
--repository-path="${path_to_repo}" \
--api-definition-path="${api_definition_path}"
--api-definitions-path="${api_definitions_path}"
```


Expand Down Expand Up @@ -157,7 +157,7 @@ docker run \
-v /path/to/google-cloud-java:/workspace \
-v /path/to/api-definition:/workspace/apis \
$(cat image-id) \
--api-definition-path=/workspace/apis
--api-definitions-path=/workspace/apis
```

* `-u "$(id -u)":"$(id -g)"` makes docker run the container impersonating
Expand All @@ -169,7 +169,7 @@ docker run \
* `-v /path/to/api-definition:/workspace/apis` maps the host machine's
api-definition folder to /workspace/apis folder.
* `$(cat image-id)` obtains the image ID created in the build step.
* `--api-definition-path=/workspace/apis` set the API definition path to
* `--api-definitions-path=/workspace/apis` set the API definition path to
`/workspace/apis`.

## Debug the created containers
Expand Down
6 changes: 3 additions & 3 deletions library_generation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ right version for each library.
Please refer [here](go/java-client-releasing#versionstxt-manifest) for more info
of versions.txt.

### Api definition path (`api_definition_path`), optional
### Api definitions path (`api_definitions_path`), optional

The path to where the api definition (proto, service yaml) resides.

Expand All @@ -61,7 +61,7 @@ Any missing dependencies will cause `File not found` error.
For example, if your service is defined in `example_service.proto` and it imports
`google/api/annotations.proto`, you need the `annotations.proto` resides in a
folder that has the exact structure of the import statement (`google/api` in this
case), and set `api_definition_path` to the path contains the root folder (`google`
case), and set `api_definitions_path` to the path contains the root folder (`google`
in this case).

## Output of `entry_point.py`
Expand Down Expand Up @@ -219,7 +219,7 @@ python library_generation/entry_point.py generate \
--baseline-generation-config-path=/path/to/baseline_config_file \
--current-generation-config-path=/path/to/current_config_file \
--repository-path=path/to/repository \
--api-definition-path=path/to/api_definition
--api-definitions-path=path/to/api_definition
```
If you run `entry_point.py` with the example [configuration](#an-example-of-generation-configuration)
shown above, the repository structure is:
Expand Down
14 changes: 7 additions & 7 deletions library_generation/cli/entry_point.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ def main(ctx):
""",
)
@click.option(
"--api-definition-path",
"--api-definitions-path",
type=str,
default=".",
show_default=True,
Expand All @@ -78,7 +78,7 @@ def generate(
baseline_generation_config_path: str,
current_generation_config_path: str,
repository_path: str,
api_definition_path: str,
api_definitions_path: str,
):
"""
Compare baseline generation config and current generation config and
Expand All @@ -105,15 +105,15 @@ def generate(
baseline_generation_config_path=baseline_generation_config_path,
current_generation_config_path=current_generation_config_path,
repository_path=repository_path,
api_definition_path=api_definition_path,
api_definitions_path=api_definitions_path,
)


def __generate_repo_and_pr_description_impl(
baseline_generation_config_path: str,
current_generation_config_path: str,
repository_path: str,
api_definition_path: str,
api_definitions_path: str,
):
"""
Implementation method for generate().
Expand Down Expand Up @@ -145,15 +145,15 @@ def __generate_repo_and_pr_description_impl(

current_generation_config_path = os.path.abspath(current_generation_config_path)
repository_path = os.path.abspath(repository_path)
api_definition_path = os.path.abspath(api_definition_path)
api_definitions_path = os.path.abspath(api_definitions_path)
if not baseline_generation_config_path:
# Execute full generation based on current_generation_config if
# baseline_generation_config is not specified.
# Do not generate pull request description.
generate_from_yaml(
config=from_yaml(current_generation_config_path),
repository_path=repository_path,
api_definition_path=api_definition_path,
api_definitions_path=api_definitions_path,
)
return

Expand All @@ -173,7 +173,7 @@ def __generate_repo_and_pr_description_impl(
generate_from_yaml(
config=config_change.current_config,
repository_path=repository_path,
api_definition_path=api_definition_path,
api_definitions_path=api_definitions_path,
target_library_names=target_library_names,
)
generate_pr_descriptions(
Expand Down
6 changes: 3 additions & 3 deletions library_generation/generate_repo.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
def generate_from_yaml(
config: GenerationConfig,
repository_path: str,
api_definition_path: str,
api_definitions_path: str,
target_library_names: list[str] = None,
) -> None:
"""
Expand All @@ -35,7 +35,7 @@ def generate_from_yaml(
:param config: a GenerationConfig object.
:param repository_path: The repository path to which the generated files
will be sent.
:param api_definition_path: The path to where the api definition resides.
:param api_definitions_path: The path to where the api definition resides.
:param target_library_names: a list of libraries to be generated.
If specified, only the library whose library_name is in target_library_names
will be generated.
Expand All @@ -49,7 +49,7 @@ def generate_from_yaml(
gen_config=config, library_config=target_libraries, repo_path=repository_path
)
# copy api definition to output folder.
shutil.copytree(api_definition_path, repo_config.output_folder, dirs_exist_ok=True)
shutil.copytree(api_definitions_path, repo_config.output_folder, dirs_exist_ok=True)

for library_path, library in repo_config.get_libraries().items():
print(f"generating library {library.get_library_name()}")
Expand Down
16 changes: 8 additions & 8 deletions library_generation/test/cli/entry_point_unit_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,12 +105,12 @@ def test_generate_non_monorepo_without_changes_triggers_full_generation(
baseline_generation_config_path=config_path,
current_generation_config_path=config_path,
repository_path=".",
api_definition_path=".",
api_definitions_path=".",
)
generate_from_yaml.assert_called_with(
config=ANY,
repository_path=ANY,
api_definition_path=ANY,
api_definitions_path=ANY,
target_library_names=None,
)

Expand Down Expand Up @@ -138,12 +138,12 @@ def test_generate_non_monorepo_with_changes_triggers_full_generation(
baseline_generation_config_path=baseline_config_path,
current_generation_config_path=current_config_path,
repository_path=".",
api_definition_path=".",
api_definitions_path=".",
)
generate_from_yaml.assert_called_with(
config=ANY,
repository_path=ANY,
api_definition_path=ANY,
api_definitions_path=ANY,
target_library_names=None,
)

Expand All @@ -168,12 +168,12 @@ def test_generate_monorepo_with_common_protos_triggers_full_generation(
baseline_generation_config_path=config_path,
current_generation_config_path=config_path,
repository_path=".",
api_definition_path=".",
api_definitions_path=".",
)
generate_from_yaml.assert_called_with(
config=ANY,
repository_path=ANY,
api_definition_path=ANY,
api_definitions_path=ANY,
target_library_names=None,
)

Expand All @@ -199,11 +199,11 @@ def test_generate_monorepo_without_common_protos_does_not_trigger_full_generatio
baseline_generation_config_path=config_path,
current_generation_config_path=config_path,
repository_path=".",
api_definition_path=".",
api_definitions_path=".",
)
generate_from_yaml.assert_called_with(
config=ANY,
repository_path=ANY,
api_definition_path=ANY,
api_definitions_path=ANY,
target_library_names=[],
)
8 changes: 4 additions & 4 deletions library_generation/test/integration_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ def setUp(cls) -> None:
os.makedirs(f"{golden_dir}", exist_ok=True)

def test_entry_point_running_in_container(self):
api_definition_path = self.__copy_api_definition(googleapis_commitish)
api_definitions_path = self.__copy_api_definition(googleapis_commitish)
config_files = self.__get_config_files(config_dir)
for repo, config_file in config_files:
config = from_yaml(config_file)
Expand All @@ -92,7 +92,7 @@ def test_entry_point_running_in_container(self):
config_location=config_location,
baseline_config=baseline_config_name,
current_config=current_config_name,
api_definition=api_definition_path,
api_definition=api_definitions_path,
)
# 4. compare generation result with golden files
print(
Expand Down Expand Up @@ -191,7 +191,7 @@ def test_entry_point_running_in_container(self):
)
print(" PR description comparison succeed.")
self.__remove_generated_files()
shutil.rmtree(api_definition_path)
shutil.rmtree(api_definitions_path)

@classmethod
def __copy_api_definition(cls, committish: str) -> str:
Expand Down Expand Up @@ -324,7 +324,7 @@ def __run_entry_point_in_docker_container(
image_tag,
f"--baseline-generation-config-path=/workspace/config/{baseline_config}",
f"--current-generation-config-path=/workspace/config/{current_config}",
f"--api-definition-path=/workspace/api",
f"--api-definitions-path=/workspace/api",
],
)

Expand Down
Loading