Explore issue with automated transcript generation

[tleonhardt]

Currently the ability to automatically generate a transcript with history -t transcript_file.txt works well, but not perfectly. There are a minority of cases where it generates a transcript which nominally looks correct but which fails actual transcript testing.

The automated transcript generation feature is awesome if users can count on it 100% of the time (or at least very close to it). So we need to figure out what is going wrong in the minority of cases. Perhaps we need to do something slightly different when generating the transcript. Or perhaps, transcript testing could be a little bit more permissive, particularly towards differences in trailing whitespace at the end of a line, end of a command, and end of the file.

The one case I am certain of that creates a bad transcript is:

help history
history -t transcript.txt

Edit: The title and text changed to reflect that we need to figure out what is going on and to try to avoid jumping to a conclusion in regards to a potential solution.

[kotfu]

I completely agree that the new history -t function to generate a transcript should always produce a transcript which will pass transcript testing. Do you have an example that doesn't work?

[tleonhardt]

I found a few examples last night that didn't work, I'll see if I can replicate one ...

[tleonhardt]

Overall the history -t functionality seems pretty reliable. The cases I can find where it doesn't work properly so far appear to exclusively be because something is getting printed to sys.stdout or sys.stderr instead of self.stdout or because some command behaves slightly differently when outputting to a pipe/file instead of a terminal.

Lets just keep this issue around for a little while until we have more experience using the history -t functionality in a wider variety of circumstances.

In some cases maybe we can make small changes to make a few more things go to self.stdout, but in other cases it is unavoidable. Like if somebody runs a shell command, we don't have full control over that. Or if they write a Python script and run it with pyscript and they put a print() in there, we can't prevent that either.

[tleonhardt]

@kotfu Ok, I finally have a decent example for you. Using the latest code on the sub-commands branch (in the PR), if you do the following in a cmd2 application:

./cmd2.py
help history
history -t transcript.txt
quit

And then when you attempt to validate the generated transcript, it will fail:

./cmd2.py -t transcript.txt

[kotfu]

The problem is that the help text for the history command includes slashes. According to our current transcript rules, anything enclosed in slashes is a regular expression. In the case of the history help, we don't intend for the thing inside the slashes to be a regular expression. That's why it fails. If you change the /regex/ literal in the help to |regex|, then the transcript works fine.

I think we can fix this by changing the history transcript generation code to escape any slashes in the output. If you are generating a transcript from history, you probably don't intend for there to be regular expressions in the transcript. This proposed fix, however, will not be an easy one to make, because the history transcript code does it's work by setting self.stdout to the opened transcript file. Not easy to intercept the byte stream and escape the slashes.

I'm gonna think about this a bit more and see if I can come up with another, less difficult approach to solve it.

[tleonhardt]

I can see a couple possible solutions other than the one you mentioned:

1. For the history command, replace the usage of "/" as the regex escape character with "|". This is a quick and easy. But it feels a little asymmetric since we use "/" as the regex escape for transcripts.

2. Add a post-processing step to history -t where after the file has been written, we grep through it and escape any "/" characters which are not already escaped. This is a little bit more work, but still be relatively easy since it is done after the file is completely written so we don't have to intercept the byte stream. It should also have the benefit of working with any arbitrary command a user creates for their application.

On a side note, what would you think about making the regex-escape character configurable, for example as an attribute within cmd.Cmd? I can imagine some applications where "/" is pretty commonly used - anything involving paths on macOS or Linux for example.