Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Port output-check-record feature to the nrunner #3882

Open
beraldoleal opened this issue Jun 4, 2020 · 4 comments
Open

Port output-check-record feature to the nrunner #3882

beraldoleal opened this issue Jun 4, 2020 · 4 comments
Assignees
@beraldoleal
Labels
customer:Passt Requirements/issues raised by the Passt project legacy
Milestone
109 - Codename TBD

Comments

@beraldoleal
Copy link
Member

beraldoleal commented Jun 4, 2020 •
edited
Loading

The idea of the features --output-check-record and --output-check are valid and we have today a few use cases that explore those functionalities.

The "problem" with this logic is that it is being used also to generate the test-results/stdout, stderr, and output files, depending on the --output-check-record value. I can understand that we are using this logic today because we are limited by the subprocess.Popen() ability to generate both "combined" and "split" (stdout and stderr) and also to avoid calling the tests twice.

We could split this logic into 3:

  1. Sometimes the users would like to run the avocado only to generate the ".data/" folder with stdout, stderr, and combined results. For this situation we could have a boolean option named --record-output and the code can be executed twice to accomplish that. This run is something like a "dry-run" and there is no need to save information on job-results folder;

  2. A normal run that MUST be executed only once that will generate the job-results information and will not care about the --record-output

  3. A run that will check for the expected values stored inside .data/. This could be done with something like --check-output, also a boolean variable that will use the same logic that we have today:

3.1 If output.expected is there, then run the subprocess generating a "combined" version;
3.2 if stdout.expected or stderr.expected are there, then run the subprocess generating a "both/split" version;

@clebergnu any comments?
@beraldoleal beraldoleal added the nrunner (previously nrun2run) label Jun 4, 2020
@beraldoleal beraldoleal self-assigned this Jun 4, 2020
@clebergnu
Copy link
Contributor

@beraldoleal I think you've captured it all. Nothing to add for now.

@beraldoleal
Copy link
Member Author

@clebergnu I'm trying to focus on issues related to our effort to make nrun feature complete.

I have added this label "nrun2run" here, but I'm thinking better now, and I'm not sure if this is a dependency for the Next Runner, right?

I just would like to double-check with you if we can remove this label "nrun2run" here.

@richtja richtja added this to the #85 milestone Dec 22, 2020
@beraldoleal beraldoleal modified the milestones: #85 (Bacurau), #86 (tbd) Feb 10, 2021
@clebergnu clebergnu modified the milestones: #86 (The Dig), TDB Mar 15, 2021
@clebergnu clebergnu modified the milestones: #87 (Braveheart), #88 (TBD) Apr 12, 2021
@clebergnu clebergnu modified the milestones: #89 (Shrek), #90 (TBD) Jun 21, 2021
@richtja richtja modified the milestones: #90 (Blade Runner), #91(TBD) Jul 26, 2021
@beraldoleal beraldoleal modified the milestones: #91(TBD), #92 (TBD) Jul 26, 2021
clebergnu added a commit to clebergnu/avocado that referenced this issue Aug 27, 2021
@clebergnu
runner switch: pin tests that are specific to the "runner" (legacy)
488a650 
Many existing tests cover features that are specific to the legacy
runner, and it makes no or little sense to port them to be compatible
with the nrunner.

Examples include:

 * selftests/functional/test_basic.py:RunnerOperationTest.test_unsupported_status
  - The legacy runner contains extra handling after the test process
    ends, and this test depends on that behavior

 * selftests/functional/test_basic.py:RunnerOperationTest.test_hanged_test_with_status
 * selftests/functional/test_basic.py:RunnerOperationTest.test_no_status_reported
  - Relies on Avocado trying to kill test processes, only performed in
    the legacy runner

 * selftests/functional/test_output_check.py:
  - Output check feature is limited to the legacy and data files
    available along side the test
  - avocado-framework#3882

And many other similar examples.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
clebergnu added a commit to clebergnu/avocado that referenced this issue Aug 27, 2021
@clebergnu
runner switch: pin tests that are specific to the "runner" (legacy)
edb3438 
Many existing tests cover features that are specific to the legacy
runner, and it makes no or little sense to port them to be compatible
with the nrunner.

Examples include:

 * selftests/functional/test_basic.py:RunnerOperationTest.test_unsupported_status
  - The legacy runner contains extra handling after the test process
    ends, and this test depends on that behavior

 * selftests/functional/test_basic.py:RunnerOperationTest.test_hanged_test_with_status
 * selftests/functional/test_basic.py:RunnerOperationTest.test_no_status_reported
  - Relies on Avocado trying to kill test processes, only performed in
    the legacy runner

 * selftests/functional/test_output_check.py:
  - Output check feature is limited to the legacy and data files
    available along side the test
  - avocado-framework#3882

And many other similar examples.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
clebergnu added a commit to clebergnu/avocado that referenced this issue Aug 27, 2021
@clebergnu
runner switch: pin tests that are specific to the "runner" (legacy)
742055f 
Many existing tests cover features that are specific to the legacy
runner, and it makes no or little sense to port them to be compatible
with the nrunner.

Examples include:

 * selftests/functional/test_basic.py:RunnerOperationTest.test_unsupported_status
  - The legacy runner contains extra handling after the test process
    ends, and this test depends on that behavior

 * selftests/functional/test_basic.py:RunnerOperationTest.test_hanged_test_with_status
 * selftests/functional/test_basic.py:RunnerOperationTest.test_no_status_reported
  - Relies on Avocado trying to kill test processes, only performed in
    the legacy runner

 * selftests/functional/test_output_check.py:
  - Output check feature is limited to the legacy and data files
    available along side the test
  - avocado-framework#3882

And many other similar examples.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
clebergnu added a commit to clebergnu/avocado that referenced this issue Aug 27, 2021
@clebergnu
runner switch: pin tests that are specific to the "runner" (legacy)
5de7faf 
Many existing tests cover features that are specific to the legacy
runner, and it makes no or little sense to port them to be compatible
with the nrunner.

Examples include:

 * selftests/functional/test_basic.py:RunnerOperationTest.test_unsupported_status
  - The legacy runner contains extra handling after the test process
    ends, and this test depends on that behavior

 * selftests/functional/test_basic.py:RunnerOperationTest.test_hanged_test_with_status
 * selftests/functional/test_basic.py:RunnerOperationTest.test_no_status_reported
  - Relies on Avocado trying to kill test processes, only performed in
    the legacy runner

 * selftests/functional/test_output_check.py:
  - Output check feature is limited to the legacy and data files
    available along side the test
  - avocado-framework#3882

And many other similar examples.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
clebergnu added a commit to clebergnu/avocado that referenced this issue Aug 27, 2021
@clebergnu
runner switch: pin tests that are specific to the "runner" (legacy)
e0609c2 
Many existing tests cover features that are specific to the legacy
runner, and it makes no or little sense to port them to be compatible
with the nrunner.

Examples include:

 * selftests/functional/test_basic.py:RunnerOperationTest.test_unsupported_status
  - The legacy runner contains extra handling after the test process
    ends, and this test depends on that behavior

 * selftests/functional/test_basic.py:RunnerOperationTest.test_hanged_test_with_status
 * selftests/functional/test_basic.py:RunnerOperationTest.test_no_status_reported
  - Relies on Avocado trying to kill test processes, only performed in
    the legacy runner

 * selftests/functional/test_output_check.py:
  - Output check feature is limited to the legacy and data files
    available along side the test
  - avocado-framework#3882

And many other similar examples.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
clebergnu added a commit to clebergnu/avocado that referenced this issue Aug 27, 2021
@clebergnu
runner switch: pin tests that are specific to the "runner" (legacy)
c02eeac 
Many existing tests cover features that are specific to the legacy
runner, and it makes no or little sense to port them to be compatible
with the nrunner.

Examples include:

 * selftests/functional/test_basic.py:RunnerOperationTest.test_unsupported_status
  - The legacy runner contains extra handling after the test process
    ends, and this test depends on that behavior

 * selftests/functional/test_basic.py:RunnerOperationTest.test_hanged_test_with_status
 * selftests/functional/test_basic.py:RunnerOperationTest.test_no_status_reported
  - Relies on Avocado trying to kill test processes, only performed in
    the legacy runner

 * selftests/functional/test_output_check.py:
  - Output check feature is limited to the legacy and data files
    available along side the test
  - avocado-framework#3882

And many other similar examples.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
beraldoleal pushed a commit to beraldoleal/avocado that referenced this issue Aug 28, 2021
@clebergnu @beraldoleal
runner switch: pin tests that are specific to the "runner" (legacy)
a5508f6 
Many existing tests cover features that are specific to the legacy
runner, and it makes no or little sense to port them to be compatible
with the nrunner.

Examples include:

 * selftests/functional/test_basic.py:RunnerOperationTest.test_unsupported_status
  - The legacy runner contains extra handling after the test process
    ends, and this test depends on that behavior

 * selftests/functional/test_basic.py:RunnerOperationTest.test_hanged_test_with_status
 * selftests/functional/test_basic.py:RunnerOperationTest.test_no_status_reported
  - Relies on Avocado trying to kill test processes, only performed in
    the legacy runner

 * selftests/functional/test_output_check.py:
  - Output check feature is limited to the legacy and data files
    available along side the test
  - avocado-framework#3882

And many other similar examples.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
beraldoleal pushed a commit to beraldoleal/avocado that referenced this issue Aug 28, 2021
@clebergnu @beraldoleal
runner switch: pin tests that are specific to the "runner" (legacy)
be2017c 
Many existing tests cover features that are specific to the legacy
runner, and it makes no or little sense to port them to be compatible
with the nrunner.

Examples include:

 * selftests/functional/test_basic.py:RunnerOperationTest.test_unsupported_status
  - The legacy runner contains extra handling after the test process
    ends, and this test depends on that behavior

 * selftests/functional/test_basic.py:RunnerOperationTest.test_hanged_test_with_status
 * selftests/functional/test_basic.py:RunnerOperationTest.test_no_status_reported
  - Relies on Avocado trying to kill test processes, only performed in
    the legacy runner

 * selftests/functional/test_output_check.py:
  - Output check feature is limited to the legacy and data files
    available along side the test
  - avocado-framework#3882

And many other similar examples.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
@clebergnu clebergnu modified the milestones: #92 Monsters, Inc., #93 Sep 23, 2021
clebergnu added a commit to clebergnu/avocado that referenced this issue Apr 20, 2022
@clebergnu
Legacy runner: remove unused "output-check-record feature" code
3e9135e 
The "output-check-record" feature was removed from the legacy runner a
while ago.  This seems to be unused code that could have been removed
then, so better be removed now.

Reference: avocado-framework#5185
Reference: avocado-framework#3882
Signed-off-by: Cleber Rosa <crosa@redhat.com>
clebergnu added a commit to clebergnu/avocado that referenced this issue Apr 21, 2022
@clebergnu
Legacy runner: remove unused "output-check-record feature" code
e2168f3 
The "output-check-record" feature was removed from the legacy runner a
while ago.  This seems to be unused code that could have been removed
then, so better be removed now.

Reference: avocado-framework#5185
Reference: avocado-framework#3882
Signed-off-by: Cleber Rosa <crosa@redhat.com>
clebergnu added a commit to clebergnu/avocado that referenced this issue Apr 22, 2022
@clebergnu
Legacy runner: remove unused "output-check-record feature" code
9f8f966 
The "output-check-record" feature was removed from the legacy runner a
while ago.  This seems to be unused code that could have been removed
then, so better be removed now.

Reference: avocado-framework#5185
Reference: avocado-framework#3882
Signed-off-by: Cleber Rosa <crosa@redhat.com>
clebergnu added a commit to clebergnu/avocado that referenced this issue Apr 26, 2022
@clebergnu
Legacy runner: remove unused "output-check-record feature" code
ec2b4c1 
The "output-check-record" feature was removed from the legacy runner a
while ago.  This seems to be unused code that could have been removed
then, so better be removed now.

Reference: avocado-framework#5185
Reference: avocado-framework#3882
Signed-off-by: Cleber Rosa <crosa@redhat.com>
@richtja richtja moved this to Long Term (Next Q) Backlog in Default project Dec 15, 2022
@clebergnu clebergnu self-assigned this Jan 23, 2023
@richtja richtja added this to the #102 (Codename TBD) milestone Mar 15, 2023
@richtja richtja moved this from Long Term (Next Q) Backlog to Short Term (Current Q) Backlog in Default project Mar 15, 2023
@richtja richtja moved this from Short Term (Current Q) Backlog to In progress in Default project Apr 26, 2023
@clebergnu
Copy link
Contributor

I have a BluePrint under development that will propose this feature under the new Avocado architecture (nrunner). Stay tuned!

@clebergnu clebergnu moved this from In progress to Short Term (Current Q) Backlog in Default project Jul 13, 2023
@clebergnu clebergnu added the customer:Passt Requirements/issues raised by the Passt project label Mar 27, 2024
@clebergnu clebergnu added this to the #105 - Codename TBD milestone Mar 27, 2024
@clebergnu clebergnu removed their assignment Oct 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@beraldoleal beraldoleal

Labels
customer:Passt Requirements/issues raised by the Passt project legacy
Projects
Default project
Status: Short Term (Current Q) Backlog
Milestone
109 - Codename TBD
Development

No branches or pull requests
4 participants
@beraldoleal @clebergnu @richtja @harvey0100