Skip to content

Regression Testing #2#189

Merged
tzanio merged 16 commits intomasterfrom
regression-auto-test-dev
Aug 19, 2021
Merged

Regression Testing #2#189
tzanio merged 16 commits intomasterfrom
regression-auto-test-dev

Conversation

@publixsubfan
Copy link
Copy Markdown
Contributor

This PR cleans up the commit history from #176.

Summary

  • Adds CTest-based regression testing.
    • A test driver, tests/glvis_driver.py, runs glvis with a modified stream that generates screenshots.
      Two screenshots are generated: one with the original image from the stream, and one with the * key command applied to induce a slight change in the image.
      Screenshots can then be compared against a baseline with the structural similarity metric. A given test case passes if the original image is above the SSIM cutoff of 0.999 and the control image is below this cutoff.
    • Tests are run in headless mode for both ubuntu-20.04 and macos-10.15 in Github Actions, with streams and corresponding baselines available in GLVis/data
  • Bugfixes/miscellaneous changes:
    • Fix some issues with screenshots on macOS and Linux
    • Improve SignalKeyDown() handling for X11
    • Handle resizes synchronously in the main thread
    • Improve error handling with SDL

- On Linux with the Mesa software renderer (and potentially other
  OpenGL drivers such as Intel's), taking screenshots from the front
  buffer is not supported. To fix this, we instead do an expose to
  ensure that the correct image is present on the back buffer, then
  capture the frame data from the back buffer.

- On Mac OS X, a call to [NSOpenGLContext update] may be required before
  doing any rendering. Instead of GLVisCommand directly calling for a
  screenshot capture, we signal for a screenshot to be taken once
  GLVisCommand::Execute() returns.
- Bypass SDL event queue for key events signalled on an SdlWindow via
  SignalKeyDown().
  Instead, we can push them directly into the window-specific event
  queue and wake the worker thread.
- Synchronously handle certain main thread events, such as resizes.
- If SDL initialization failed, don't start the main event thread, and
  return an invalid handle from createWindow() to terminate any worker
  threads.
- Call SDL_Quit() in SdlMainThread's destructor to cleanup SDL at exit.
- GLEW may sometimes return an error on Wayland, since it can't find an
  X display. However, this is not a fatal error, and OpenGL functions
  can still be initialized, so we can just continue.
- It appears Mesa creates some thread-local data for each OpenGL
  context, which is then freed upon thread exit. This seems to conflict
  with the cleanup mechanisms in SDL; as a workaround, we unbind the
  context on the worker thread before deleting on the main thread.
Copy link
Copy Markdown
Member

@tzanio tzanio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much @kanye-quest !

@tzanio
Copy link
Copy Markdown
Member

tzanio commented Aug 19, 2021

Can you please resolve the conflicts (from #190)?

Copy link
Copy Markdown
Contributor

@adrienbernede adrienbernede left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!
I tested it locally and it worked like a charm after settings things up (I have a python environment for glvis-testing).
I felt the need to insist on a bit on the developer workflow (make rebaseline) in the baseline details section.
Feel free to accept or reject.

publixsubfan and others added 2 commits August 19, 2021 15:34
Co-authored-by: Adrien Bernede <51493078+adrienbernede@users.noreply.github.com>
@tzanio
Copy link
Copy Markdown
Member

tzanio commented Aug 19, 2021

@kanye-quest, any ideas why the builds / macos-10.15-optim-sequential-cmake test is failing?

Edit: I swear I saw failures, but now the tests look good :)

@tzanio
Copy link
Copy Markdown
Member

tzanio commented Aug 19, 2021

@adrienbernede, can we change the GH actions names to be consistent with the new ones in mfem?

@tzanio tzanio merged commit 25fcbfb into master Aug 19, 2021
@tzanio tzanio deleted the regression-auto-test-dev branch August 19, 2021 23:53
@tzanio
Copy link
Copy Markdown
Member

tzanio commented Aug 20, 2021

I can't express how cool this new regression testing system is... thanks so much @kanye-quest !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants