Skip to content

Add generated protos #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cretz merged 8 commits into from
Jun 9, 2022
Merged

Add generated protos #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1b0fdc2
More tests, docs, and minor fixes
cretz Jun 3, 2022
f3f6962
More workflow tests
cretz Jun 6, 2022
75d0e48
Work on docs
cretz Jun 6, 2022
21b4f2d
Merge remote-tracking branch 'remotes/origin/main' into more-tests
cretz Jun 7, 2022
8dad0fe
Minor PR update
cretz Jun 7, 2022
245cb8a
Add generated protobuf files
cretz Jun 7, 2022
5c0c4c1
Merge remote-tracking branch 'remotes/origin/main' into protos
cretz Jun 9, 2022
d7d69f7
Fix leftover merge artifact
cretz Jun 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ jobs:
- os: ubuntu-latest
python: 3.10
docsTarget: true
protoCheckTarget: true
Copy link
Member

@bergundy bergundy Jun 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did this in TS and TBH it's probably better to just have a separate build for docs and proto check to avoid delaying the entire pipeline.

Copy link
Member Author

@cretz cretz Jun 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fast enough, and it's on the fastest target so still gets done before mac/windows. Proto gen is quite fast.

runs-on: ${{ matrix.os }}
steps:
- name: Print build information
Expand All @@ -41,6 +42,14 @@ jobs:
- run: poe build-develop
- run: poe test -s -o log_cli_level=DEBUG

# Confirm protos are already generated properly
- name: Check generated protos
if: ${{ matrix.protoCheckTarget }}
run: |
poe gen-protos
poe format
[[ -z $(git status --porcelain) ]] || (git status; echo "Protos changed"; exit 1)

# Do docs stuff (only on one host)
- name: Build API docs
if: ${{ matrix.docsTarget }}
Expand Down Expand Up @@ -91,7 +100,6 @@ jobs:
go-version: "1.17"
- run: python -m pip install --upgrade wheel poetry poethepoet
- run: poetry install --no-root
- run: poe gen-protos
- run: poetry build
- run: poe fix-wheel
- run: poe test-dist-single
Expand Down
4 changes: 0 additions & 4 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,6 @@
__pycache__
/build
/dist
temporalio/api/*
!temporalio/api/__init__.py
temporalio/bridge/proto/*
!temporalio/bridge/proto/__init__.py
temporalio/bridge/target/
temporalio/bridge/temporal_sdk_bridge*
/tests/helpers/golangserver/golangserver
Expand Down
7 changes: 0 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -595,12 +595,6 @@ it):
poetry install --no-root
```

Now generate the protobuf code:

```bash
poe gen-protos
```

#### Build

Now perform the release build:
Expand Down Expand Up @@ -680,7 +674,6 @@ installing dependencies, and generating the protobuf code:
git clone --recursive https://github.com/temporalio/sdk-python.git
cd sdk-python
poetry install --no-root
poe gen-protos
```

Now compile the Rust extension in develop mode which is quicker than release mode:
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ toml = "^0.10.2"
twine = "^3.8.0"

[tool.poe.tasks]
build-develop = ["gen-protos", "build-bridge-develop"]
build-develop = ["build-bridge-develop"]
build-bridge-develop = "python scripts/setup_bridge.py develop"
fix-wheel = "python scripts/fix_wheel.py"
format = [{cmd = "black ."}, {cmd = "isort ."}]
Expand Down
11 changes: 10 additions & 1 deletion scripts/gen_protos.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,13 +68,22 @@ def fix_generated_output(base_path: Path):
f.write(content)
# Write init
with (base_path / "__init__.py").open("w") as f:
# Add docstring to API's init
if str(base_path.as_posix()).endswith("/temporal/api"):
f.write('"""gRPC API."""\n')
Copy link
Member

@bergundy bergundy Jun 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why?

Copy link
Member Author

@cretz cretz Jun 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we fail my poe lint-docs test because it's a package with no API. Notice right now at https://python.temporal.io/ we don't have a summary doc for just the temporal.api package. That's because I had traditionally gen'd docs after protos (which is after the doc linter) and didn't have this here.

# Imports
message_names = []
for stem, messages in imports.items():
for message in messages:
f.write(f"from .{stem} import {message}\n")
# MyPy protobuf does not document this experimental class, see
# https://github.com/nipunn1313/mypy-protobuf/issues/212#issuecomment-885300106
import_suffix = ""
if stem == "service_pb2_grpc" and message == "WorkflowService":
import_suffix = " # type: ignore"
f.write(f"from .{stem} import {message}{import_suffix}\n")
message_names.append(message)
# __all__
message_names = sorted(message_names)
if message_names:
f.write(
f'\n__all__ = [\n "' + '",\n "'.join(message_names) + '",\n]\n'
Expand Down
Empty file added temporalio/api/command/__init__.py
View file
Open in desktop
Empty file.
33 changes: 33 additions & 0 deletions temporalio/api/command/v1/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
from .message_pb2 import (
Copy link
Member Author

@cretz cretz Jun 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every file starting here is auto-generated

CancelTimerCommandAttributes,
CancelWorkflowExecutionCommandAttributes,
Command,
CompleteWorkflowExecutionCommandAttributes,
ContinueAsNewWorkflowExecutionCommandAttributes,
FailWorkflowExecutionCommandAttributes,
RecordMarkerCommandAttributes,
RequestCancelActivityTaskCommandAttributes,
RequestCancelExternalWorkflowExecutionCommandAttributes,
ScheduleActivityTaskCommandAttributes,
SignalExternalWorkflowExecutionCommandAttributes,
StartChildWorkflowExecutionCommandAttributes,
StartTimerCommandAttributes,
UpsertWorkflowSearchAttributesCommandAttributes,
)

__all__ = [
"CancelTimerCommandAttributes",
"CancelWorkflowExecutionCommandAttributes",
"Command",
"CompleteWorkflowExecutionCommandAttributes",
"ContinueAsNewWorkflowExecutionCommandAttributes",
"FailWorkflowExecutionCommandAttributes",
"RecordMarkerCommandAttributes",
"RequestCancelActivityTaskCommandAttributes",
"RequestCancelExternalWorkflowExecutionCommandAttributes",
"ScheduleActivityTaskCommandAttributes",
"SignalExternalWorkflowExecutionCommandAttributes",
"StartChildWorkflowExecutionCommandAttributes",
"StartTimerCommandAttributes",
"UpsertWorkflowSearchAttributesCommandAttributes",
]
Loading