Skip to content

Conversation

@fzhinkin
Copy link
Collaborator

Also, updated notebooks to be more flexible with params.

@fzhinkin fzhinkin requested a review from cmelchior October 30, 2025 20:24
@fzhinkin
Copy link
Collaborator Author

@cmelchior could you please help with notebooks: I was going to add yet another notebook that'll compare benchmarks from the same run with each other and stumbled across a problem I can't solve quickly.

The JSON file with run results may not contain the param attribute for benchmarks, and in this case I don't understand how to inject some default value (like null), so that the rest of a notebook will continue to work without any changes.

For instance, consider plugging such a result file into a notebook: https://gist.githubusercontent.com/fzhinkin/afb9423170056af2696877f837f7950b/raw/31f6bb2e45ba5888f0b78ad7ebb9e62d5be563b0/main.json

Maybe you have some good ideas?

@cmelchior
Copy link
Contributor

Hmm, it can be tricky to support benchmarks with and without params. The reason is that our JSON type generator uses the JSON content it has available to generate types. So if there are no params in the JSON, the params property doesn't get generated.

I see two solutions to this:

  1. We have two notebooks, one with param support and one without.
  2. We embed the schema files in the notebook and use them for deserialization. This will make the notebook slightly larger, but it means it can be used for both types of tests with no modifications.

I am leaning towards solution 2 and can add the modification. If you agree?

For the example file you had there, I copied a notebook and did minimal adjustments to remove it, trying to parse the params:
https://gist.github.com/cmelchior/84e4f13a20e7204202d63b2ee4f151ec

@fzhinkin
Copy link
Collaborator Author

@cmelchior, the second option sounds good to me. After all, later all that could be moved to a separate artifacts that notebooks will use as a dependency, I guess.

For the example file you had there, I copied a notebook and did minimal adjustments to remove it, trying to parse the params:
https://gist.github.com/cmelchior/84e4f13a20e7204202d63b2ee4f151ec

It seems like it does not work :(

@cmelchior
Copy link
Contributor

I pushed updates to the example notebooks that should make them work in a generic manner.

The key change is using custom serialization classes that uses JsonObject for the param. This way, we can treat those in a dynamic manner in the rest of the notebook.

The serialization classes can be created using "jsonText".deserializeThis().getCode(), but would then create a Param class that matches whatever file is being used.

@fzhinkin fzhinkin requested a review from ilya-g November 12, 2025 23:27
@fzhinkin
Copy link
Collaborator Author

Slightly trimmed the Benchmark DTO, so it now works with both JMH format and a simplified version of this format used for other targets.

Also, added an example project comparing some "baseline" and "optimized" implementations of the same algorithm, and added a notebook that analyses its results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants