Skip to content

Add Replay ("Policy") performance tests (TaskCompletionRateTest) #704

Open
@abrichr

Description

@abrichr

Feature request

We need to extend #314 to include some useful tests and generate an automated report.

This involves:

  • Create recordings of three tasks:
  1. Open a calculator and perform a short calculation
  2. Open a spreadsheet (e.g. https://github.com/OpenAdaptAI/OpenAdapt/blob/cb70f35985eeb579fd3e13b20a9839b10729921d/tests/assets/excel.png), open a time tracking app (e.g. https://clockify.me), copy a week's worth of data from the spreadsheet into the app, and save/submit the data in the app. (e.g. https://www.youtube.com/watch?v=omP11q-o_0I)
    Alternatively if browser events are not yet available (see Add Chrome browser event in database during recording #744), replicate something similar with two different spreadsheets open simultaneously (one for reading, one for writing).
  3. Open powerpoint and create a short presentation.
  • Save them as fixtures
  • Add automated tests to run a replay (with configurable strategy, defaulting to VanillaReplayStrategy) and evaluate the outcome. Outcome evaluation can be implemented with WindowEvent data.
  • Add a script to log the outcome results to stdout and/or to a file.

Motivation

Scientific rigor and reproducibility.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions