-
Notifications
You must be signed in to change notification settings - Fork 203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Java/CSharp/Go - Basic usage of MockServer in IT #2410
Conversation
Converting to draft as, I think, this can be expanded to be a bit more generic and reusable already. |
@baywet this is still missing a good developer experience (the mockserver stays alive in background ATM), but I would be happy to receive early feedback. I moved all the logic to PowerShell scripts so that the IT test can simply assume that a server is running and it can be easily implemented in any lang. wdyt? |
4e809ab
to
caee921
Compare
ok, I'm happy enough with this as an iteration:
ready for review! |
Thanks for the extensive work here. First, I'm not sure how much value we're getting from this setup without significant investments: we need to handcraft the API calls for each language for each API description. I'm not sure how to address that at the moment. We (the microsoft graph client experience team) do have automated integration testing that makes the API calls in a project called raptor. But that came at a price of years of work to:
A lot of that stack is dependent on OData aspects and older technologies today, and there are no plans to upgrade the stack or make it more generic at this point. Then, pulling a random jar from the web and running the code without "any trace of the dependency" seems like a risky approach from a supply chain perspective and being able to trace security issues. |
Thanks, as always, for taking the time to review this PR @baywet !
Here I was picking up from this comment and I do believe that is a good step forward for the project even in the current state. Short-term (e.g. this PR):
Mid-term: Expanding(even manually!) those tests to cover dynamically typed languages is going to dramatically improve the consistency as(I believe) there is, currently, little value from the "compilation" and "linting" phase we are performing. Long-term: Automating the generation of the tests is a super interesting subject, I think it makes sense in a project like this to test the generator itself and the integration with core libraries. All in all, I understand your worries, but I believe that the benefits in the short/mid-term are dramatically outweighing the possible disadvantages of(mismanaged) longer-term plans.
this is not accurate:
That said:
As said, I have to disagree with the motivation 🙂, but I have no strong objection to going down this path if requested. Thanks for powering through this long answer. 🙂 |
Thanks for the detailed answer. Goals: this clarification helps. So in your view we'd only implement tests for a few endpoints of a few descriptions? Security: I wasn't specific enough. Our security and compliance have tooling checking dependencies for any given project to let us know whether we might be at risk. Let's say tomorrow a vulnerability is discovered in mock server 5.3, if that dependency is listed in a "configuration file" (github workflows, csproj, pom, etc...) we'll get automated alerts and PRs to bump things up. With a PowerShell script like that, it's a bit more opaque to them. Do you think we could instead have a configuration relying on a POM for the mock server and have that path in dependabot? |
This is the goal of this PR, extensive coverage and further development expanding the current scope need to be discussed.
Yes, sure, I can configure Maven to fetch the dependency so that it will be tracked and updated with dependabot. Do we have an agreement? 🤝 🙂 |
Yes as long as you are willing to add tests for a few other languages as part of this PR (dotnet and Go would be a good start) |
ok, deal, I move this back to draft until I have wrapped up everything. |
caee921
to
0aa2316
Compare
Co-authored-by: Vincent Biret <vibiret@microsoft.com>
Co-authored-by: Vincent Biret <vibiret@microsoft.com>
049ee42
to
5bd6b67
Compare
rebased on latest |
CI is green ✅ 🎉 |
@baywet thanks for merging! |
To recap what was said on the meeting for anybody else watching that space. Yes we will over time, however we'll wait a little bit to make sure the integration tests are stable before doing so. This is to avoid blocking people should those tests require further investments. |
Thanks for the recap @baywet ! |
This PR spearheads the usage of MockServer to eventually achieve full coverage with the IT tests.
Note:
Here the choice of Maven is completely deliberate as it's much faster for batch executions, and also the integration with the MockServer is extremely well tightened.
(Been there here: exercism/java-test-runner#37 exercism/java-test-runner#39)