Find a way to group debugging output with relevant test case

[bdw429s]

In CFML, you can output directly to the console like so:

systemOutput( 'my debug message', true )

This is crude but effective, however if you have a test suite with a lot of tests, all the debugging messages show up at the top. I noticed in the JS katas, calls to console.log() are grouped within each spec's output. CFML provides no way to capture the output from this function as it is fed directly to the System.out stream in the JVM.

I have a few ideas, but I'd need to think about how we're calling the tests to see what's available. TestBox itself has a debug() method and it tracks a debug stream for each spec which is available in the test results. However, the debug method is only available inside the actual test cases, not in the solution CFC. One option is we "inject" a debug method by default into the solution component at runtime for users to call which would be "spec aware".

Another option may be to look into the echo() function in CFML. It writes content to a print buffer that exists in each thread. (In a web context, this print buffer is what becomes the output to the browser). It's possible to call a UDF in CFML and wrap it in a manner that captures all output from that method:

function myMethodThatOutputsStuff() {
  echo( 'I like spam' )
}

savecontent variable='local.out' {
  myMethodThatOutputsStuff()
}
// Variable local.out contains text "I like spam"

This seems promising as echo() is a CF built-in wouldn't require any special injections into the Solution cfc, however since the actual spec closures are executed inside of the TestBox framework, I'm not sure Codewar's runner would have the opportunity to wrap each one. Actually-- come to think of it-- we may be able to do this with a base test case all our test cases extend that uses an aroundEach() advice to wrap the execution of each spec, feeding any output into the debug stream for that test. 🤔

https://testbox.ortusbooks.com/primers/testbox-bdd-primer/life-cycle-methods#around-specs

component extends="testbox.system.BaseSpec" {
     /**
     * @aroundEach
     */
	 function captureOutput( spec, suite ){
		savecontent variable="local.out" {
    		arguments.spec.body();
		}
		if( len( trim( local.out ) ) ) {
			debug( local.out )
		}
 	 }
}

We could put that ☝️ in a base spec and then change the codewars specs to all extend that class so they inherit the aroundeach behavior.

---

Let me know your thoughts and how I can help experiment with this. I'm sure there's a better solution we can find, but I'm not positive about all the plumbing in place nor how to play around and test with it.

[bdw429s]

I forgot to add above, both of those suggestions also assume that the CLI runner for CFML would be reworked to capture the debug stream fromTestBox and work it into the output of the results page. The debug stream is an array of objects, wherein each object has a "label" that is a string matching the nested suite and spec names. So for example, if you have a bundle containing the following suite and spec:

describe( "my suite", function(){
  it( "my spec", function(){
    debug( 'My message' )
    expect(...).toBe(...);
  });
});

Then the corresponding debug message in the array would have a label of

/my suite/my spec

Keep in mind, suites can nest as deep as you like so it may be

/my suite/my nested suite/yet another nested suite/my spec

I wish there was a little better way of matching debug messages. If it were me, I'd do a single pass over the array and transform it into a struct/object of arrays where the key was the unique label.
Then while outputting the spec results, you'd need to recursively track the spec and all ancestor suite labels to check for any corresponding debug messages in your lookup object.

The good news is, I'm also a committer to the TestBox project, so I can help improve this in future versions of the TestBox core to serve our purposes better :)

[kazk]

Let me know your thoughts and how I can help experiment with this. I'm sure there's a better solution we can find, but I'm not positive about all the plumbing in place nor how to play around and test with it.

box install can install packages from GitHub right? I can package CodewarsReporter and TestRunner and accept contributions.

If we can make it work with box task run TestRunner, it'll work on Codewars.

[bdw429s]

Yes, box install supports everything npm install does and probably some more. That's a good idea to have it in a repo. So long as there was a clear way to clone them locally and play, that would be fine.

https://commandbox.ortusbooks.com/package-management/code-endpoints/git#github-shortcut

So

install user/reo

or

install user/repo#commitish

A package in CommandBox has a box.json not unlike a package.json works. You can initialize package in a directory with package init. (You would want the git repo itself to be a package)
https://commandbox.ortusbooks.com/package-management/creating-packages
The main box.json that listed the repo as a dependency can also use the installpaths to override where it gets put.
https://commandbox.ortusbooks.com/package-management/box.json/installation#installpaths
And there are some other options that can go in the package itself to help control how it gets installed:
https://commandbox.ortusbooks.com/package-management/box.json/installation#createpackagedirectory

[kazk]

Created https://github.com/Codewars/testbox-codewars

I wasn't sure what's needed in box.json so I didn't include it. I'd like to be able to install that repo using box install and then run box task run TestRunner.

[kazk]

In order to display the logs under test cases, the output must be in between <IT::> and the test result (<PASSED::>, <FAILED::>, etc).

https://github.com/Codewars/testbox-codewars/blob/a830ef7ef6e0e4d4c43afeadc2e1807fdeed34e0/CodewarsReporter.cfc#L43-L45

It doesn't need any special prefixes.

[bdw429s]

Excellent! Looking at this now.

Just to get a feel for how you want this to be handled. I can add the box.json to the repo and have it include testbox as a dependency. Then you can just do

box install Codewars/testbox-codewars

And it will grab everything in this repo and testbox too. I assume that all happens when you build the image, and you copy the test case and solution file in when the container deploys?

The only remaining question would be if it matters where the files end up. The default behavior for CommandBox installing a package that's of a generic type (not a module, etc) would be for it to simply go into a folder in the current working directory named after the package's slug. So you'd end up with something like ./testbox-codewars/TestRunner.cfc etc. If that's fine, then no problem. If you want to change the name of the folder, or you want it to not go in a subfolder at all, just let me know and I'll adjust the box.json appropriately.

[kazk]

I assume that all happens when you build the image, and you copy the test case and solution file in when the container deploys?

Yes, all box install happens when the image is built. Then we create a container, put files under /workspace, then start the container with box task run TestRunner.

The only remaining question would be if it matters where the files end up.

So if we do what you described and did box install Codewars/testbox-codewars we'll have the following files:

/workspace/testbox-codewars/CodewarsReporter.cfc
/workspace/testbox-codewars/TestRunner.cfc
/workspace/Solution.cfc
/workspace/SolutionTest.cfc

Then we can just do box task run testbox-codewars/TestRunner?

This should be fine.

[bdw429s]

Umm, that changes a couple paths, but it can work. The only place the subfolder would be noticeable to users is each test suite would need to extend testbox-codewars.CodewarsBaseSpec instead of just extending CodewarsBaseSpec. Unlike languages like Java, where class packages are sort of arbitrary and compiled into them, CFML uses relative directory paths to define a components path. So a folder structure of /foo/bar/Baz.cfc would be created as new foo.bar.Baz(). That's why the folder name would show up in the component path.

[kazk]

It does? The working directory is /workspace.

[kazk]

Can you set it up the way you think is natural? We can try to make that work.

[bdw429s]

Yes, but it doesn't start at the drive root. Technically it starts at the "web root" since CFML was first designed for web apps, but there's no web root in the CLI so I digress. CF has several rules for resolving component paths.

• Can it be found in the web root (this is a moot rule for our CLI use case)
• can it be found relative to the current folder of the calling code
• Can it be found in any CF mappings (we'll ignore this one for practical purposes. An example of this is the code where I create a CF mapping for /testbox. That's because the testbox core framework expects testbox.system.foo to resolve to the folder where it live.)

Anywhoo, the containing folder of workspace really has no bearing here. The task runner in question is where the relative paths start when CF goes looking for a CFC path using its default rules. So the word workspace would not appear in the path, it would start in that folder and go down.

[kazk]

Maybe we're misunderstanding? /workspace shouldn't be significant. It's just a working directory where the files are written to.

So the word workspace would not appear in the path, it would start in that folder and go down.

Yeah, that's what I thought. Maybe I confused you when I listed the files using absolute paths in my comment.