-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Split check_test_cases.py and outcome_analysis.py #9668
Split check_test_cases.py and outcome_analysis.py #9668
Commits on Oct 3, 2024
-
Create a module to split test case collection from checks
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for 1e5dbd3 - Browse repository at this point
Copy the full SHA 1e5dbd3View commit details -
Split test case collection from checks
Move the test case collection code out of check_test_cases.py and into its own module. This allows outcome analysis to depend only on the new module and not on check_test_cases.py. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for c8c83d0 - Browse repository at this point
Copy the full SHA c8c83d0View commit details -
Create a module to split branch-independent code out of analyze_outco…
…mes.py Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for 9f930e0 - Browse repository at this point
Copy the full SHA 9f930e0View commit details -
Remove sample ignore list elements for coverage
The ignore list for coverage only has two test cases out of ~10000 that are currently reported as not executed. This is a drop in the sea and not useful. Remove them so that the class can be used generically. A follow-up will construct a comprehensive ignore list. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for 49c77dd - Browse repository at this point
Copy the full SHA 49c77ddView commit details -
Missing NotImplementedError in abstract method
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for 9d78e87 - Browse repository at this point
Copy the full SHA 9d78e87View commit details -
Don't reuse a variable name inside a function
Use different names for task name, a task class and a task instance. The interpreter doesn't care, but it's less confusing for both humans and type checkers. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for ad02d44 - Browse repository at this point
Copy the full SHA ad02d44View commit details -
Always have tasks_list be a list, not potentially some fancier iterable. Bypass mypy's somewhat legitimate complaint about REFERENCE and DRIVER in task_class: they could potentially be instance attributes, but we rely on them being class attributes. Python does normally guarantee their existence as class attributes (unless a derived class explicitly deletes them), but they could be overridden by an instance attribute; that's just something we don't do, so the class attribute's value is legitimate. We can't expect mypy to know that, so work around its complaint. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for 005dca6 - Browse repository at this point
Copy the full SHA 005dca6View commit details -
Pass KNOWN_TASKS as an argument to main
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for e41cde5 - Browse repository at this point
Copy the full SHA e41cde5View commit details -
Separate code and data of outcome analysis
Place the code of outcome analysis (auxiliary functions, tasks, command line entry point) into a separate module, which will be moved to the version-independent framework repository so that it can be shared between maintained branches. Keep the branch-specific list of driver components and ignore lists in the per-repository script. We keep the executable script at `tests/scripts/analyze_outcomes.py`. It's simpler that way, because that path is hard-coded in CI scripts. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for 082eade - Browse repository at this point
Copy the full SHA 082eadeView commit details
Commits on Oct 8, 2024
-
Adjust paths for impending moves to the framework
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for 3146772 - Browse repository at this point
Copy the full SHA 3146772View commit details
Commits on Oct 9, 2024
-
Move test case analysis modules to framework repository
Move `collect_test_cases.py` (split from `check_test_cases.py`), `check_test_cases.py`, and `outcome_analysis.py` (split from `analyze_outcomes.py`) to the framework repository. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for 9f85399 - Browse repository at this point
Copy the full SHA 9f85399View commit details -
Upgrade mypy to the last version supporting Python 3.6
Upgrade mypy to 0.971, which is the last version that supports Python 3.6 (the oldest Python version that we currently run on the CI). This fixes the error ``` framework/scripts/mbedtls_framework/outcome_analysis.py:119: error: Incompatible return value type (got "IO[Any]", expected "TextIO") framework/scripts/mbedtls_framework/outcome_analysis.py:121: error: Incompatible return value type (got "IO[Any]", expected "TextIO") ``` As far as I can tell the fix is python/mypy#9275 which was released in mypy 0.940. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for 1c5a252 - Browse repository at this point
Copy the full SHA 1c5a252View commit details -
Downgrade mypy to a version that works with our code base
mypy >=0.960 rejects macro_collector.py. Mbed-TLS/mbedtls-framework#50 We currently need mypy >=0.940, <0.960. Pick 0.942, which works, and is the system version on Ubuntu 22.04. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for 041a84d - Browse repository at this point
Copy the full SHA 041a84dView commit details -
Default to allowing partial test coverage
Currently, many test cases are not executed. A follow-up pull request will take care of that. In the meantime, continue allowing partial test coverage. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for 96db2cc - Browse repository at this point
Copy the full SHA 96db2ccView commit details
Commits on Oct 10, 2024
-
Update framework to the branch with collect_test_cases.py and outcome…
…_analysis.py Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Configuration menu - View commit details
-
Copy full SHA for 8fa4964 - Browse repository at this point
Copy the full SHA 8fa4964View commit details