Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: improved results directory structure for load generation #24869

Merged
merged 4 commits into from
Apr 2, 2024

Conversation

hiltontj
Copy link
Contributor

@hiltontj hiltontj commented Apr 2, 2024

Closes #24831

write and query load generation runners will now automatically setup files in a results directory, using a pre-defined structure. Users of the load tool can specify a results_dir to save these results, or the tool will pick a results folder in the current directory, by default.

Results will be saved in files using the following path convention:

<results_dir>/<s>/<c>/<write|query|system>_<time>.csv
  • <results_dir>: the specified results_dir, or the results/ folder in the current directory
  • <s>: spec name
  • <c>: configuration name, specified by user with the config-name arg, or by default, will use the revision SHA of the running server
  • <write|query|system>: which kind of results file
  • <time>: a timestamp in the form YYYY-MM-DD-HH-MM

The resulting file structure will look like this:

results
├── one_mil
│  └── 270f721fd7
│     ├── query_2024-04-02-12-21.csv
│     ├── system_2024-04-02-12-21.csv
│     └── write_2024-04-02-12-19.csv
└── sample_spec
   ├── 270f721fd7
   │  └── write_2024-04-02-12-24.csv
   └── my-load-test
      └── write_2024-04-02-12-25.csv

The setup code was unified for both write and query commands, in preparation for the creation of a system stats file, as well as for the capability to run both query and write at the same time, however, those tasks remain for future PRs.

Other Changes

  • Added /ping API support to the influxdb3_client::Client

hiltontj added 3 commits April 2, 2024 09:34
Added the results_dir and configuration_name args
to the common load generator config which will be
used in generating the results directory structure.
Write and query load generation runners will now setup files in a
results directory, using a specific structure. Users of the load tool
can specify a `results_dir` to save these results, or the tool will
pick a `results` folder in the current directory, by default.

Results will be saved in files using the following path convention:

results/<s>/<c>/<write|query|system>_<time>.csv

- <s>: spec name
- <c>: configuration name, specified by user with the `config-name`
  arg, or by default, will use the revision SHA of the running server
- <write|query|system>: which kind of results file
- <time>: a timestamp in the form 'YYYY-MM-DD-HH-MM'

The setup code was unified for both write and query commands, in
preparation for the creation of a system stats file, as well as for
the capability to run both query and write at the same time, however,
those remain unimplemented as of this commit.
Copy link
Member

@pauldix pauldix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great stuff 🚢

};
// if print spec is set, print the spec and exit
if print_spec {
bail!("exiting after printing spec");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL

@hiltontj
Copy link
Contributor Author

hiltontj commented Apr 2, 2024

@pauldix - I pushed up a commit that added some doc comments, but nothing functionally different, so once this is ✅ I will merge

@hiltontj hiltontj merged commit cc55685 into main Apr 2, 2024
9 checks passed
@hiltontj hiltontj deleted the th/load-gen-results branch April 2, 2024 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update load generator to have results directory
2 participants