add command line flag for system test result dumps

[efd6]

Please take a look.

[mrodm]

Adding just one flag it forces to always set a prefix to the user, but probably they often need just a default value for that.

What about adding two flags to manage this feature?

• One flag to enable or not dumping the documents (--dump-docs) without defining any path or prefix.
  • The default filename could be the same one used when the environment variable is set.
  • Should this file be saved in the same location where elastic-package runs ? Take into account that this could change with the parameter -C.

    elastic-package -C test/packages/parallel/nginx test system -v

• Another flag to set a custom prefix (--dump-docs-prefix) or even it could be set the whole path (--dump-docs-output) leaving to the user the choice to where save the output file. I think I'd prefer the latter.

# enable dumping documents and using the default filename for the path
elastic-package test system -v --dump-docs-

# enable dumping documents and using a custom path
elastic-package test system -v --dump-docs --dump-docs-output /path/to/dump.json

Doing it like this, it would be similar on how coverage is set in elastic-package:

• --test-coverage to enable the feature,
• --coverage-format to choose which format is required.

[efd6]

I think this comes down to a philosophical position; I think having a default location for things like this is a significant misfeature. IMO actions should require explicit requests. Including things like this in the config add complexity to the user experience and is also IMO a misfeature.

I'm not entirely sure what the gain is for users in adding this additional complexity.

[mrodm]

IMO I think most users (from my POV) would like just to download/dump the documents without really requiring setting any specific path or prefix. That's why I would say that the most common usage would be (not requiring to set any prefix there):

elastic-package test system -v --dump-docs

Moreover, doing it like this, it would enable this feature and it could be coded to be equivalent to set the ELASTIC_PACKAGE_TEST_DUMP_SCENARIO_DOCS environment variable. So users setting the environment variable and setting that flag can expect the same exact behavior of elastic-package.

For the other flag I proposed, I thought there could be users that require some specific known file, for instance they are running tests with different packages and they want those files to be easily identified. That's my reasoning to add that new --dump-docs-output or --dump-docs-prefix flag, so they can decide how to save those documents.

# allow a full path
elastic-package test system -v --dump-docs --dump-docs-output /path/store/documents.json

# or as you proposed adding a prefix and save them in the same location where elastic-package runs
elastic-package test system -v --dump-docs --dump-docs-prefix "package-test"

The concern I have of writing those new documents in the same location as elastic-package runs is that this could lead to users committing and pushing those files in the repository inadvertently.

[efd6]

IMO I think most users (from my POV) would like just to download/dump the documents without really requiring setting any specific path or prefix.

This is certainly not what I'd want; doing this means that I cannot script obtaining the data.

In terms of work, specifying a location is easier than digging through logs to find the name that the program chose. In my ideal world, the output would not be to a file whose name was under the control of the program at all. IMO this should be entirely under the control of the user, but I made a concession to the current behaviour on the basis that it was still relatively easy to obtain the files if the prefix was known.

[mrodm]

Ok, got it.

Could it be changed what the value would indicate for this flag? Instead of setting a prefix for files, this flag could set the path to a folder.

I'm trying to be consistent with other commands/parameters used in elastic-package. For instance, in these commands --output is a path to a folder:

• dump stack logs: elastic-package stack dump --output <path>
• dump assets: elastic-package dump agent-policies --output <path>

That folder should be checked (and created if needed) by elastic-package. And it would be the folder where the JSON files are stored.
Files stored in that folder could follow the same naming patterns as it was set when setting the environment variable:

    dumpPath = filepath.Join(dumpFolderBase, fmt.Sprintf("elastic-package-test-docs-dump-%s.json", time.Now().Format("20060102150405")))

So, elastic-package would be like this:

elastic-package test system -v --dump-logs "docs_folder_package/"

Probably, doing it like this, the parameter could be renamed as --dump-logs-to.

[efd6]

I'm trying to be consistent with other commands/parameters used in elastic-package.

ISTM that to be consistent with those the flag should be called dump.

That folder should be checked (and created if needed) by elastic-package.

I think in many areas ep does too much (I'm a strong follower of the unix philosophy), creating paths is a thing that is done by mkdir.

elastic-package test system -v --dump-logs "docs_folder_package/"

I think that I'd be OK with stating the path and determining whether it's a directory and using this behaviour in that case, otherwise using the behaviour that exists in this change. Note that the dumped artefacts are not logs in the general case, so naming the flag that would be misleading/confusing (I'm already thinking that "dump" is a very long spelling of "o").

[mrodm]

Note that the dumped artefacts are not logs in the general case, so naming the flag that would be misleading/confusing (I'm already thinking that "dump" is a very long spelling of "o").

Oops, I was wrong in the comment above, sorry. I wanted to write for the flag names these two options: --dump-docs and --dump-docs-to.

I'm trying to be consistent with other commands/parameters used in elastic-package.

ISTM that to be consistent with those the flag should be called dump.

Using just dump for the flag name, I see that it could lead to confusions in the context of the elastic-package test system command. For me, it's not clear what it is going to dump: Elastic Agent logs, errors found, documents ingested into ES, mappings...

elastic-package test system -v --dump-logs "docs_folder_package/"

I think that I'd be OK with stating the path and determining whether it's a directory and using this behaviour in that case, otherwise using the behaviour that exists in this change.

I don't think offering different behaviours depending on the value of the same flag would be helpful here. I'd rather to be always the same behaviour and deterministic, and not depending if the string set in the flag matches with a folder or not in the filesystem.

@elastic/ecosystem could you chime in here too? I'd like to know your thoughts for this new flag too, thanks!

[jsoriano]

@elastic/ecosystem could you chime in here too? I'd like to know your thoughts for this new flag too, thanks!

I think I'd also prefer this flag to specify a folder instead of a prefix. I would avoid implementing it in a way that behaves differently depending on the syntax of the parameter.

This is certainly not what I'd want; doing this means that I cannot script obtaining the data.

@efd6 by knowing the directory it would be also possible to script obtaining the data, right?

I think in many areas ep does too much (I'm a strong follower of the unix philosophy) creating paths is a thing that is done by mkdir.

It definitely does many things 🙂 elastic-package is more a toolkit than a unix tool, also intended to work on different operating systems, including Windows. It can be seen as a set of scripts, and even in the unix world, scripts create directories, by calling mkdir.

[efd6]

I think I'd also prefer this flag to specify a folder instead of a prefix.

Which behaviour would you prefer:

1. mkdir -p ${outdir}
   echo ${result} > ${outdir}/${prefix_slug}-$(date).ndjson
   

2. mkdir ${outdir}
   echo ${result} > ${outdir}/${prefix_slug}-$(date).ndjson
   

(where ${prefix_slug} is determined without user input and $(data) is the current timestamp suffix logic).

even in the unix world, scripts create directories, by calling mkdir.

In those cases they mkdir not mkdir -p. This is the distinction that I was making.

[jsoriano]

I think I'd also prefer this flag to specify a folder instead of a prefix.

Which behaviour would you prefer.

1. mkdir -p ${outdir}
   echo ${result} > ${outdir}/${prefix_slug}-$(date).ndjson
   

2. mkdir ${outdir}
   echo ${result} > ${outdir}/${prefix_slug}-$(date).ndjson
   

(where ${prefix_slug} is determined without user input and $(data) is the current timestamp suffix logic).

I prefer the first behavior, as this is more convenient for users.

This is also what elastic-package does at the moment in most of the cases, so it would be consistent with current behaviors:

.../elastic-package$ git grep Mkdir | wc -l
65
.../elastic-package$ git grep MkdirAll | wc -l
60
.../elastic-package$ git grep MkdirTemp | wc -l
4

Though it is also true that maybe scripts don't always try to create parent directories, so for this case both options would work for me if you prefer the second behavior.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: eb7be7f

History

• 💔 Build #6274 failed 9456b6f
• 💚 Build #6266 succeeded a9fe9ba
• 💔 Build #6265 failed cd7a5e3

cc @efd6