Java/CSharp/Go - Basic usage of MockServer in IT

[andreaTP]

This PR spearheads the usage of MockServer to eventually achieve full coverage with the IT tests.

Note:
Here the choice of Maven is completely deliberate as it's much faster for batch executions, and also the integration with the MockServer is extremely well tightened.
(Been there here: exercism/java-test-runner#37 exercism/java-test-runner#39)

[andreaTP]

Converting to draft as, I think, this can be expanded to be a bit more generic and reusable already.

[andreaTP]

@baywet this is still missing a good developer experience (the mockserver stays alive in background ATM), but I would be happy to receive early feedback.

I moved all the logic to PowerShell scripts so that the IT test can simply assume that a server is running and it can be easily implemented in any lang.

wdyt?

[andreaTP]

ok, I'm happy enough with this as an iteration:

• there is only one test running the MockServer
• all of the infrastructure is already prepared
• it should be relatively easy to extend and integrate more tests/languages

ready for review!

[baywet]

Thanks for the extensive work here.
I do have a couple of thoughts on the matter.

First, I'm not sure how much value we're getting from this setup without significant investments: we need to handcraft the API calls for each language for each API description. I'm not sure how to address that at the moment. We (the microsoft graph client experience team) do have automated integration testing that makes the API calls in a project called raptor. But that came at a price of years of work to:

1. generate http snippets from the API description
2. generate code snippet from the http snippets
3. orchestrate those snippets into buildable code
4. setup the code so it can run (auth, etc...)

A lot of that stack is dependent on OData aspects and older technologies today, and there are no plans to upgrade the stack or make it more generic at this point.

Then, pulling a random jar from the web and running the code without "any trace of the dependency" seems like a risky approach from a supply chain perspective and being able to trace security issues.
I'd favor running it in a container to reduce the potential damage that could be done in case the package gets compromised.

[andreaTP]

Thanks, as always, for taking the time to review this PR @baywet !
Let me expand on the context for this PR:

First, I'm not sure how much value we're getting from this setup without significant investments

Here I was picking up from this comment and I do believe that is a good step forward for the project even in the current state.

Short-term (e.g. this PR):

• easily write integration tests to reproduce runtime issues(like this one)
• be able to share reproducers for runtime issues with a standardized setup
• write "smoke tests" to verify that a PR doesn't (accidentally) breaks core functionalities of the project
• write regression tests for overall functionalities
• dev speed in bug fixing having a "standard" setup

Mid-term:

Expanding(even manually!) those tests to cover dynamically typed languages is going to dramatically improve the consistency as(I believe) there is, currently, little value from the "compilation" and "linting" phase we are performing.
Expanding those tests to the languages that have already reached GA is going to give us good guarantees over the fact that there are no accidental breaking changes(which is suitable for enabling contributions from the community etc.).

Long-term:

Automating the generation of the tests is a super interesting subject, I think it makes sense in a project like this to test the generator itself and the integration with core libraries.
I agree that writing "test generators" might be a significant effort, but this is a step in the right direction as an enabler.

All in all, I understand your worries, but I believe that the benefits in the short/mid-term are dramatically outweighing the possible disadvantages of(mismanaged) longer-term plans.
At the same time, we are enabling further development in this direction for the future which makes me think that is a good direction.

---

pulling a random jar from the web and running the code without "any trace of the dependency"

this is not accurate:

• we are downloading from Maven Central official servers using the https protocol, which guarantees the identity of the server, so there is no security risk for supply chain attacks, verifying the hash(sha, md5) of the downloaded package would improve but it's common practice to trust the default certificate authority(open to doing the change if requested though)
• we are using "one of" the officially documented ways of running MockServer
• the package we are using is specifically designed for distribution and is guaranteed to contain all of the needed dependencies (in Java they call it a uber-jar in this case with shaded dependencies), basically is the equivalent of a statically linked binary for the JVM

That said:

I'd favor running it in a container to reduce the potential damage that could be done in case the package gets compromised.

As said, I have to disagree with the motivation 🙂, but I have no strong objection to going down this path if requested.
The tradeoffs I considered when proposing the jar approach are the dependency on a Container runtime (Docker, Podman etc.), which is currently not a requirement, and the speed of execution.

---

Thanks for powering through this long answer. 🙂
Happy to discuss this further if is needed!

[baywet]

Thanks for the detailed answer.

Goals: this clarification helps. So in your view we'd only implement tests for a few endpoints of a few descriptions?

Security: I wasn't specific enough. Our security and compliance have tooling checking dependencies for any given project to let us know whether we might be at risk. Let's say tomorrow a vulnerability is discovered in mock server 5.3, if that dependency is listed in a "configuration file" (github workflows, csproj, pom, etc...) we'll get automated alerts and PRs to bump things up. With a PowerShell script like that, it's a bit more opaque to them. Do you think we could instead have a configuration relying on a POM for the mock server and have that path in dependabot?

[andreaTP]

So in your view we'd only implement tests for a few endpoints of a few descriptions?

This is the goal of this PR, extensive coverage and further development expanding the current scope need to be discussed.
The short answer is Yes.

Security

Yes, sure, I can configure Maven to fetch the dependency so that it will be tracked and updated with dependabot.
I considered the risk pretty low here as the package is used only as infrastructure on GH Action to run the IT tests(e.g. no artifact is produced and exposed to users in those pipelines), but fine for me 👍 .

Do we have an agreement? 🤝 🙂

[baywet]

Yes as long as you are willing to add tests for a few other languages as part of this PR (dotnet and Go would be a good start)

[andreaTP]

ok, deal, I move this back to draft until I have wrapped up everything.

[andreaTP]

rebased on latest main, let's see how this goes...

[andreaTP]

CI is green ✅ 🎉

[andreaTP]

@baywet thanks for merging!
After this fix things should be more stable 🙂 would you consider adding the IT tests passing as a requirement to merge PRs?

[baywet]

To recap what was said on the meeting for anybody else watching that space. Yes we will over time, however we'll wait a little bit to make sure the integration tests are stable before doing so. This is to avoid blocking people should those tests require further investments.

[andreaTP]

Thanks for the recap @baywet !
Can I ask if we can keep track of any noticed "spurious failure" opening a relevant issue for it?
Feel free to assign those to me.