Skip to content

Conversation

nicolexin
Copy link
Contributor

@nicolexin nicolexin commented Apr 1, 2025

Adding initial implementer's guide (currently focusing on writing a controller), this covers:

  1. How InferencePool fits into the overall Gateway API resource model
  2. Implementation options
  3. Callout extension protocol
  4. Testing tips

@k8s-ci-robot k8s-ci-robot requested review from ahg-g and liu-cong April 1, 2025 18:27
@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 1, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @nicolexin. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 1, 2025
Copy link

netlify bot commented Apr 1, 2025

Deploy Preview for gateway-api-inference-extension ready!

Name Link
🔨 Latest commit 58600f6
🔍 Latest deploy log https://app.netlify.com/sites/gateway-api-inference-extension/deploys/67eda2cd9da8d200088b595a
😎 Deploy Preview https://deploy-preview-635--gateway-api-inference-extension.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@ahg-g
Copy link
Contributor

ahg-g commented Apr 1, 2025

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 1, 2025
- **Using Reference EPP Implementation + Echoserver**: You can use the [reference EPP implementation](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/pkg/epp) for testing your controller end-to-end. Instead of a full-fledged model server, a simple mock server (like the echoserver) can be very useful for verifying routing to ensure the correct pod received the request.
- **Performance Test**: Run end-to-end [benchmarks](https://gateway-api-inference-extension.sigs.k8s.io/performance/benchmark/) to make sure that your inference gateway can achieve the latency target that is desired.

???+ note
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
???+ note
## Conformance Test

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to be a subsection under Testing Tips :)

@robscott
Copy link
Member

robscott commented Apr 2, 2025

Thanks @nicolexin!

/cc @LiorLieberman @danehans

@liu-cong
Copy link
Contributor

liu-cong commented Apr 2, 2025

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 2, 2025
@danehans
Copy link
Contributor

danehans commented Apr 3, 2025

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danehans, nicolexin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 3, 2025
@k8s-ci-robot k8s-ci-robot merged commit 2a48131 into kubernetes-sigs:main Apr 3, 2025
7 checks passed
@nicolexin nicolexin deleted the implementers-guide branch April 7, 2025 22:24
kfswain pushed a commit to kfswain/llm-instance-gateway that referenced this pull request Apr 29, 2025
* Add initial implementer's guide

* Add line break to fix the list formatting

* Add line break to fix the list formatting

* Address code review comments

* Fix formatting for conformance tests
kfswain pushed a commit to kfswain/llm-instance-gateway that referenced this pull request Apr 29, 2025
* Add initial implementer's guide

* Add line break to fix the list formatting

* Add line break to fix the list formatting

* Address code review comments

* Fix formatting for conformance tests
kfswain pushed a commit to kfswain/llm-instance-gateway that referenced this pull request Apr 29, 2025
* Add initial implementer's guide

* Add line break to fix the list formatting

* Add line break to fix the list formatting

* Address code review comments

* Fix formatting for conformance tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants