Golden File Testing

[quimpm]

Would be nice to make JUnit able to support Golden File Testing.

Golden File Testing is an excellent approach for testing huge output files, API responses... It's useful when, apart from testing all the logic of your program (this is done by unit testing itself), furthermore you also want to know that what you are outputting and giving to a client, downstream, whatever... it's fine and contains what it should. Maybe in that impasse of writing the data into a file, something is not working properly.

I thought that maybe it can be done by adding an optional when calling to JUnit like --accept in order to write the result of the test to the golden file and, when that option isn't set, execute normal golden test behavior.

I thought also about different solutions like doing a library apart from JUnit and then use the approach inside unit tests. But there's always the dependency with JUnit due to the acceptance parameter when we want the test to write into the golden file. For example, if this done through an annotation in each TestCase class, every time you want to accept the output and write it to the golden files, you will have to change each TestClass... And that is a bit painful if you have a huge project with a lot of test cases.

Some links about Golden File Testing:
https://ro-che.info/articles/2017-12-04-golden-tests
https://medium.com/@jarifibrahim/golden-files-why-you-should-use-them-47087ec994bf
https://ieftimov.com/post/testing-in-go-golden-files/

[smoyer64]

Golden files are indeed useful and broadly used in Go testing. There's a great presentation by Mitchell Hashimoto (of Hashicorp) for the uninitiated and I agree that it would be useful in Java as well. If it's not part of JUnit, perhaps it can find a place in JUnitPioneer as a community supported extension.

cc: @nipafx

[sormuras]

// converted this idea into a discussion

Is Golden File Testing achievable via

• Jupiter's Testlets aka Dynamic Tests combined with Assertions.assertLinesMatch()

... or perhaps something that I was exploring at:

• https://github.com/sormuras/text-transform-tester

🤔

Except for the udpate feature creating new expected output (golden files) data on-the-fly.

[quimpm]

Yeah, as I said, I think is achievable in a lot of ways of implementation.😄 But one of the most important things in golden tests is the update feature.

And anyway, if you want to add the update feature, a strong dependency raises. This is because you will have to modify your tests code in order to make them accept the results of the execution. And that's not good. I mean, you not always can write a golden file with the expected output by hand because maybe the program has a huge output. That's why the update feature is that important for golden file testing. Now imagine you have a huge program with a lot of tests, modifying one by one the ones that you want to accept the output result and then modifying it another time to make the comparative it's painful. And writing the expected output by hand test by test is more than painful jajaja. Doing so with a flag parameter when executing the tests, it's a lot more easy.

That's why I see this highly coupled with the JUnit framework, and as a possible new feature for it.🚀

[sormuras]

Just found another instance of Golden File Testing within the JUnit 5's integration tests:

• Golden Files
  https://github.com/junit-team/junit5/tree/main/platform-tooling-support-tests/projects/jar-describe-module
• @ParameterizedTest-base implementation reading those files and comparing to lines generated by the jar tool:
  JarDescribeModuleTests.java

Let's wait and see what others think, including some @junit-team members.

[vlsi]

There's https://approvaltests.com/ for it

[4F2E4A2E]

Does something like this exist in for junit?

https://api.flutter.dev/flutter/flutter_test/GoldenFileComparator-class.html
https://pub.dev/documentation/golden_toolkit/latest/#multiscreengolden

[goatfryed]

Hey everybody,
TLDR: I released https://central.sonatype.com/artifact/io.github.goatfryed/assert-baseline a couple of weeks ago in version 0.1.0 which covers the described need.

A bit longer didn't read:

There seem to be various names for this kind of testing between golden file testing, baseline testing, snapshot testing and approval testing. Personally, I'll settle on baseline approval testing, which seems to be most broadly used and understood. Fun fact, I discovered name golden file test on an entirely unrelated topic, just read the name, checked, if it commonly used and ended up here. I would have never found this discussion otherwise, even though I rely on it heavily in e2e and data mapper code, which is why I wrote a library to support it better.

Golden file testing seems to be the least used name, if I compare github tags. Well, I added it to my project anyway. If you can educate me that it in fact better although less common, or whether there a specific differences, I'd be thankful.

Regarding my lib:

As mentioned, I released it a couple of weeks ago and have a really small user base already. I wanted to wait a bit before reaching out to the community and share it. Since people that find/follow this thread are the perfect audience, I'd be happy to receive some early feedback or ideally even some users.

Regarding @quimpm initial request about an update feature in Junit, my apporoach is quite simple. Outputs are aways written with a .actual. infix that can be git ignored. If you want to create the golden file aka baseline initially or update it, you have rename your actual file. This seems quite reasonable and I didn't fully understand the concerns about introducing a test flag. A write flag seems harder to maintain. But if this is a valid concern, i'd be happy if you could explain further, maybe even start a discussion over there.

Sorry if I hijacked the discussion here a bit for promotion, but approval baseline testing is one of my favorite topics and since I'm new to open source maintaining, I'm looking to get some reach. :)