Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Framework: Split check_test_cases.py and outcome_analysis.py #55

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
742 commits
Select commit Hold shift + click to select a range
6901504
Allow no authentication of the server in 1.3
mpg Aug 14, 2024
7a442c9
ssl-opt.sh: Fix test case titles
ronald-cron-arm Apr 3, 2024
bfbecf8
tls13: Add support for trusted certificate callback
ronald-cron-arm Apr 3, 2024
95dd6f5
ssl-opt.sh: Test trusted certificate callback in TLS 1.3
ronald-cron-arm Apr 3, 2024
5bdadbb
Restrict the scope of a few variables
mpg Aug 14, 2024
fd800c2
Improve a variable's name
mpg Aug 14, 2024
843a00d
Add support for context f_vrfy callback in 1.3
mpg Aug 16, 2024
908f57d
Minor refactoring of generic SSL certificate verif
mpg Aug 16, 2024
19dd9f5
Merge 1.2 and 1.3 certificate verification
mpg Aug 16, 2024
a040548
Improve some comments
mpg Aug 16, 2024
67072bf
Fix two dependency declarations in ssl-opt
mpg Aug 16, 2024
9e3e991
Fix typos in comments
mpg Aug 20, 2024
5398e58
Fix guards around function now used by 1.3 as well
mpg Aug 20, 2024
b721ccc
Add a ChangeLog entry
mpg Aug 20, 2024
c645f11
Merge pull request #9508 from eleuzi01/docs-fix
mpg Sep 2, 2024
c21675e
Make error line consistent with the header
eleuzi01 Sep 2, 2024
c37877f
Merge pull request #9515 from mpg/windows-makefile-fixes
mpg Sep 3, 2024
f3cd97d
Merge pull request #9327 from eleuzi01/issue-9319
mpg Sep 3, 2024
36fe918
Merge pull request #9252 from gabor-mezei-arm/9114_replace_MBEDTLS_MD…
davidhorstmann-arm Sep 3, 2024
1d98d9d
Merge pull request #9526 from mpg/refactor-tls123-verif-dev
davidhorstmann-arm Sep 3, 2024
9fc5be0
Replace MBEDTLS_MD_CAN_SHA1 with PSA_WANT_ALG_SHA_1
eleuzi01 Sep 4, 2024
7305002
Add optionally unsafe variant of exp_mod for perf
mpg Jun 18, 2024
91537eb
Improve documentation of MBEDTLS_MPI_IS_PUBLIC
yanesca Aug 12, 2024
9a8b1f4
Make _optionally_safe functions internal
yanesca Aug 12, 2024
4726cb8
Move mixed security code to small local functions
yanesca Aug 12, 2024
5b69fad
Move MBEDTLS_MPI_IS_* macros to bignum_core.h
yanesca Aug 12, 2024
24fb8c9
Make MBEDTLS_MPI_IS_PUBLIC thumb friendly
yanesca Aug 12, 2024
8fc736d
Move _public parameters next to their target
yanesca Aug 12, 2024
a099ac9
Use actual exponent size for window calculation
yanesca Aug 13, 2024
df5e55b
Add tests for optionally safe codepaths
yanesca Aug 13, 2024
1fa5f3a
Add tests for optionally unsafe code paths
yanesca Aug 13, 2024
b64f1b5
Fix mpi_core_exp_mod documentation
yanesca Aug 15, 2024
87253af
Optimise public RSA operations
yanesca Aug 15, 2024
08091d7
Fix optionally safe hooks declarations
yanesca Aug 20, 2024
d6aaee1
Disable optionally safe test hook in threading builds
yanesca Aug 20, 2024
76c0e6f
Clean up initialization in _core_exp_mod()
yanesca Aug 20, 2024
a7eb812
Fix memory corruption in exp_mod tests
yanesca Aug 20, 2024
e1d1854
Add changelog
yanesca Aug 22, 2024
6872c5f
Make mbedtls_mpi_exp_mod_unsafe internal
yanesca Aug 22, 2024
6154765
Improve ChangeLog
yanesca Aug 22, 2024
c870e05
Add header for mbedtls_mpi_exp_mod_unsafe()
yanesca Aug 22, 2024
b676959
Fix Changelog formatting
yanesca Aug 22, 2024
64467ff
Add tests for optionally safe code paths in bignum
yanesca Aug 21, 2024
e0825bb
Add tests for optionally safe code paths in RSA
yanesca Aug 21, 2024
816a71f
Introduce MBEDTLS_MPI_IS_TEST
yanesca Aug 22, 2024
47847ca
Initial local variables to secure default
yanesca Aug 22, 2024
7e909c8
Explain the choice of the value of MBEDTLS_MPI_IS_PUBLIC
yanesca Aug 22, 2024
514e62c
Move bignum code path testing out of the library
yanesca Aug 22, 2024
44eca95
Fix incorrect test result
yanesca Aug 22, 2024
21445c5
Prepare codepath tests for early termination
yanesca Aug 22, 2024
e91d924
Fix code style
mpg Sep 2, 2024
464bf78
Fix guards on #include
mpg Sep 2, 2024
3106013
Fix code style (for real this time, hopefully)
mpg Sep 2, 2024
49645f6
Misc improvements to comments
mpg Sep 3, 2024
0c4a115
Remove codepath testing where it's redundant
mpg Sep 5, 2024
b70ef86
Move new files to their correct location
mpg Sep 5, 2024
63cb13e
Replace MBEDTLS_ECP_HAVE_SECP224K1 with PSA_WANT_ECC_SECP_K1_224
eleuzi01 Sep 5, 2024
924c1a3
Merge pull request #9540 from eleuzi01/replace-sha1
davidhorstmann-arm Sep 5, 2024
1ad1954
Merge pull request #9304 from sezrab/analyze_driver_vs_reference_head…
gilles-peskine-arm Sep 5, 2024
8c95999
Merge pull request #9544 from eleuzi01/replace-224k1
ronald-cron-arm Sep 6, 2024
a9511bc
all.sh: Enable MBEDTLS_PSA_CRYPTO_CONFIG in tls13 components
ronald-cron-arm Sep 6, 2024
54d1eec
all.sh: Enable MBEDTLS_PSA_CRYPTO_CONFIG in build_aes_variations
ronald-cron-arm Sep 6, 2024
6604047
all.sh: Keep only "test_block_cipher_no_decrypt_aesni" with PSA config
ronald-cron-arm Sep 6, 2024
e7f289e
Use PSA crypto config in test_sha3_variations
ronald-cron-arm Sep 6, 2024
ada6ce5
Update framework to latest
ronald-cron-arm Sep 9, 2024
a6950b8
Replace MBEDTLS_PK_CAN_ECDSA_SOME with PSA_HAVE_ALG_SOME_ECDSA
eleuzi01 Jul 30, 2024
4fd105c
Introduce PSA_HAVE_ALG_ECDSA_SIGN and PSA_HAVE_ALG_ECDSA_VERIFY
eleuzi01 Aug 27, 2024
39c7d5d
Replace MBEDTLS_PK_CAN_ECDSA_SIGN with PSA_HAVE_ALG_ECDSA_SIGN
eleuzi01 Aug 27, 2024
bed21b5
Replace MBEDTLS_PK_CAN_ECDSA_VERIFY with PSA_HAVE_ALG_ECDSA_VERIFY
eleuzi01 Aug 27, 2024
8d8620b
Address review comments: add PSA_WANT_KEY_TYPE_ECC_KEY_PAIR_IMPORT
eleuzi01 Sep 3, 2024
96867b6
Address review comments: remove dependencies
eleuzi01 Sep 3, 2024
91d8386
Enable USE_PSA_CRYPTO
eleuzi01 Sep 4, 2024
b430eee
Bring back some dependencies
eleuzi01 Sep 5, 2024
9c64764
Remove definitions of MBEDTLS_PK_CAN/HAVE_ECDSA*
eleuzi01 Sep 6, 2024
9328781
Use scripts/project_name.txt in Python scripts
ronald-cron-arm Sep 9, 2024
7ba04a2
Bump version to 4.0.0
davidhorstmann-arm Sep 3, 2024
22eb209
Update framework
ronald-cron-arm Sep 9, 2024
ef2dd74
Run test suites with a single call to run-test-suites.pl
gilles-peskine-arm Sep 11, 2024
24515f9
Revert "Adapt basic-build-test.sh"
gilles-peskine-arm Sep 11, 2024
282abb5
Create seedfile in the tf-psa-crypto subproject
gilles-peskine-arm Sep 11, 2024
71a53b5
Fix test suites invoked in the wrong directory
gilles-peskine-arm Sep 11, 2024
25012e5
Update framework to the merge of #45
ronald-cron-arm Sep 11, 2024
4153ebb
Fix comments
ronald-cron-arm Sep 11, 2024
9fa8dc4
Fix shared library builds
gilles-peskine-arm Sep 11, 2024
236e05d
Fix typo that caused cpp_dummy_build not to be built with CMake
gilles-peskine-arm Sep 11, 2024
10ff417
tests: add a test for pkg-config files
billatarm Mar 25, 2024
8ea8b30
Merge pull request #8988 from billatarm/add-pc-files
paul-elliott-arm Sep 11, 2024
5f71537
Merge pull request #9552 from ronald-cron-arm/project-and-branches-de…
ronald-cron-arm Sep 12, 2024
2f620f3
Merge pull request #9561 from gilles-peskine-arm/cmake-underscore_D
paul-elliott-arm Sep 12, 2024
a45ba05
Merge pull request #9549 from ronald-cron-arm/adapt-components
ronald-cron-arm Sep 12, 2024
ffce45c
Add missing ALG_SHA_1
eleuzi01 Sep 12, 2024
56ee69d
Fix "Renegotiation: openssl server, client-initiated" with OpenSSL 3
gilles-peskine-arm Sep 6, 2024
9404169
Use CONFIGS_ENABLED instead of repeatedly calling query_compile_time_…
gilles-peskine-arm Sep 6, 2024
6f9952a
ssl-opt: Fix GnuTLS PSK injection
gilles-peskine-arm Sep 6, 2024
ae3dc17
Detect more cases where certificates are required
gilles-peskine-arm Sep 6, 2024
4f09864
Fix PSK-only mode doing less than it should
gilles-peskine-arm Sep 6, 2024
24b4303
Detect PSK-only mode in TLS 1.3 as well
gilles-peskine-arm Sep 6, 2024
9d3b207
Unify the two requires-key-exchange-with-certificate function
gilles-peskine-arm Sep 6, 2024
db26406
Also activate PSK-only mode when PSK-ephemeral key exchanges are avai…
gilles-peskine-arm Sep 6, 2024
cd4fe70
Fix weirdly quoted invocations of requires_any_configs_enabled
gilles-peskine-arm Sep 7, 2024
78df617
Remove unused auth_mode parameter on a PSK test case
gilles-peskine-arm Sep 7, 2024
7f453bf
Improve some comments
gilles-peskine-arm Sep 9, 2024
2dd4395
Fix detection of TLS 1.2 PSK-ephemeral key exchange modes
gilles-peskine-arm Sep 9, 2024
6e85e35
Documentation improvements
gilles-peskine-arm Sep 10, 2024
e3eab32
requires_certificate_authentication: prioritize TLS 1.3
gilles-peskine-arm Sep 10, 2024
e16aecc
Merge pull request #9556 from gilles-peskine-arm/ssl-opt-psk-detectio…
gilles-peskine-arm Sep 13, 2024
de4d5b7
Merge pull request #9497 from minosgalanakis/bugfix/return_early_in_c…
gilles-peskine-arm Sep 13, 2024
5dab92e
CMake: support "make programs"
gilles-peskine-arm Sep 14, 2024
b630648
CMake: support "make ssl-opt" to just build what ssl-opt.sh needs
gilles-peskine-arm Sep 14, 2024
d001f58
make: support "make ssl-opt" to just build what ssl-opt.sh needs
gilles-peskine-arm Sep 14, 2024
063fb8c
Move config-ccm-psk-tls1_2.h testing to separate all.sh components
gilles-peskine-arm Sep 14, 2024
68f511e
Move config-ccm-psk-dtls1_2.h testing to separate all.sh components
gilles-peskine-arm Sep 14, 2024
b6d4913
Move config-suite-b.h testing to separate all.sh components
gilles-peskine-arm Sep 14, 2024
86ceb13
Move config-thread.h testing to separate all.sh components
gilles-peskine-arm Sep 14, 2024
af5a899
Move config-symmetric-only.h testing to separate all.sh components
gilles-peskine-arm Sep 14, 2024
effa6a0
Move config-tfm.h testing to separate all.sh component
gilles-peskine-arm Sep 14, 2024
2e449f0
Remove test-ref-configs.pl, which no longer does anything
gilles-peskine-arm Sep 14, 2024
73c4096
Make the file a bit more readable
gilles-peskine-arm May 12, 2024
fbb59bd
Document the C compiler requirement
gilles-peskine-arm May 12, 2024
b805168
Merge pull request #9536 from mpg/rsapub-perf-dev
tom-cosgrove-arm Sep 18, 2024
f8db5b6
Move the function declarations to x509.h
sezrab Jul 19, 2024
4f76194
Move function mbedtls_oid_get_numeric_string to x509.c
sezrab Jul 19, 2024
4aee6a2
Move mbedtls_oid_get_numeric_string unit tests to test_suite_x509parse
sezrab Jul 19, 2024
c71abc3
Move mbedtls_oid_from_numeric_string to x509_create.c
sezrab Jul 19, 2024
2bb3f4d
Move mbedtls_oid_from_numeric_string unit tests to test_suite_x509write
sezrab Jul 19, 2024
3da783b
Move static OID functions to x509.c
sezrab Sep 13, 2024
5125a1b
Add ChangeLog for moving OID string conversion functions
Harry-Ramsey Sep 13, 2024
e5b261f
Fix ChangeLog format error
Harry-Ramsey Sep 13, 2024
9f2b817
Update documentation
gabor-mezei-arm Aug 6, 2024
c659c1b
Move config file modification flag handling to the Config class
gabor-mezei-arm Aug 6, 2024
daf807f
Fix pylint issues
gabor-mezei-arm Aug 14, 2024
d53080d
Make the `Config` a proper base class
gabor-mezei-arm Aug 27, 2024
776ee90
Fix header file detection
gabor-mezei-arm Sep 9, 2024
24d7cc7
Create a class for command line config manipulation
gabor-mezei-arm Aug 6, 2024
a12ed6b
Unify spacing
gabor-mezei-arm Sep 9, 2024
0e9e4cb
Move commonly used part to config_common
gabor-mezei-arm Sep 10, 2024
f5f1308
Fix documentation
gabor-mezei-arm Sep 18, 2024
568808a
Update member variable names
gabor-mezei-arm Sep 18, 2024
317a2a3
Fix documentation
gabor-mezei-arm Sep 18, 2024
cd326bf
Apply the parameter change
gabor-mezei-arm Sep 18, 2024
a941e14
Update framework
gabor-mezei-arm Sep 10, 2024
a22b95a
Fix obsolete comment about MBEDTLS_PSA_CRYPTO_KEY_ID_ENCODES_OWNER
gilles-peskine-arm Sep 19, 2024
bae154d
Merge pull request #9385 from eleuzi01/replace-ecdsa-some
davidhorstmann-arm Sep 19, 2024
94c3065
Fix x509 parse syntax typo
Harry-Ramsey Sep 19, 2024
e7422a1
Remove legacy-crypto components that come from test-ref-configs
gilles-peskine-arm Sep 19, 2024
ea5de2b
Simplify psa-crypto components that come from test-ref-configs
gilles-peskine-arm Sep 19, 2024
8904570
Merge pull request #9470 from gabor-mezei-arm/9325_move_config.py_to_…
ronald-cron-arm Sep 20, 2024
702d75a
Pass the setting's value to adapters
gilles-peskine-arm Sep 19, 2024
e4c6955
Change "full" to affect boolean settings rather than use sections
gilles-peskine-arm Sep 19, 2024
36571d6
Change "realfull" to activate everything
gilles-peskine-arm Sep 19, 2024
00b9144
Don't pass the section name to adapters
gilles-peskine-arm Sep 19, 2024
475ed86
Merge pull request #9600 from gilles-peskine-arm/use_psa_crypto-key_i…
tom-cosgrove-arm Sep 24, 2024
edc27bc
Merge pull request #9128 from gilles-peskine-arm/make_generated_files…
paul-elliott-arm Sep 24, 2024
26650f5
Merge pull request #9565 from gilles-peskine-arm/test-ref-configs-go-…
gilles-peskine-arm Sep 24, 2024
1d6a950
opt-testcases/*.sh are not executable
gilles-peskine-arm Sep 13, 2024
7b02c1f
Avoid multiline requires_all_configs_xxx
gilles-peskine-arm Sep 13, 2024
73a406e
Separate accepting TLS 1.3 middlebox compatibility from sending it
gilles-peskine-arm Sep 13, 2024
671a439
Adapt middlebox compatibility tests for always-on acceptance
gilles-peskine-arm Sep 13, 2024
0f2d839
Remove mid-stanza blank lines
gilles-peskine-arm Sep 13, 2024
edc8f35
Remove obsolete requirements on middlebox compatibility mode: manual
gilles-peskine-arm Sep 13, 2024
ae5a35f
Remove obsolete requirements on middlebox compatibility mode: generated
gilles-peskine-arm Sep 13, 2024
c07beda
generate_tls13_compat_tests: change default mode to all
gilles-peskine-arm Sep 13, 2024
3943a1a
Automatically generate tests/opt-testcases/tls13-compat.sh
gilles-peskine-arm Sep 13, 2024
5dd839a
Fix sensitivity of tls13-compat.sh to the exact generation method
gilles-peskine-arm Sep 13, 2024
6f03eb8
Changelog entry: fix #9551
gilles-peskine-arm Sep 13, 2024
dfbc1a9
Remove transitional always-on internal option
gilles-peskine-arm Sep 13, 2024
9ca9b92
Reduce level of non-error debug message
gilles-peskine-arm Sep 13, 2024
1e01b4a
Remove obsolete requirement for GnuTLS %DISABLE_TLS13_COMPAT_MODE
gilles-peskine-arm Sep 19, 2024
907e495
Print a "Gen" line when generating a file
gilles-peskine-arm Sep 19, 2024
40c090f
Move generation of tls13-compat.sh to tests/CMakeLists.txt
gilles-peskine-arm Sep 19, 2024
fd6d279
CMake: generate tls13-compat.sh in the default build target
gilles-peskine-arm Sep 19, 2024
1150526
Fix copypasta
gilles-peskine-arm Sep 24, 2024
63068d6
Have `make ssl-opt` generate `tls13-compat.sh`
gilles-peskine-arm Sep 24, 2024
419bacc
Allow running pylint and mypy on a single file
gilles-peskine-arm Sep 16, 2024
19ef1ae
Replace stringly typed data by class: prepare
gilles-peskine-arm Sep 16, 2024
f646dbf
Replace stringly typed data by class: coverage
gilles-peskine-arm Sep 16, 2024
82b1672
Replace stringly typed data by class: driver vs reference (code)
gilles-peskine-arm Sep 16, 2024
9df375b
Replace stringly typed data by class: driver vs reference (data)
gilles-peskine-arm Sep 16, 2024
0f31f76
Remove dead code that was handling stringly typed data
gilles-peskine-arm Sep 16, 2024
3f5022e
Move analysis functions into their respective classes
gilles-peskine-arm Sep 16, 2024
b4daeb4
Remove now-useless level of method call indirection
gilles-peskine-arm Sep 16, 2024
0297605
Move test case ignore list to the master Task class
gilles-peskine-arm Sep 16, 2024
dba8010
Simplify sub-test-suite handling in is_test_case_ignored
gilles-peskine-arm Sep 16, 2024
54cfe77
Switch coverage analysis to IGNORE_TESTS for its allowlist
gilles-peskine-arm Sep 16, 2024
870ed2a
Have `make ssl-opt` generate `tls13-compat.sh` (make edition)
gilles-peskine-arm Sep 25, 2024
2efb3da
Merge pull request #9628 from gilles-peskine-arm/tls13-middlebox-comp…
ronald-cron-arm Sep 25, 2024
6959f53
ssl_client1: Exit with an error status if the TLS connection failed
gilles-peskine-arm Aug 27, 2024
a602a41
Prepare to test SSL sample programs
gilles-peskine-arm Sep 4, 2024
2bc5c80
Allow test cases to use a specific port
gilles-peskine-arm Sep 4, 2024
7985d45
Test ssl_client1
gilles-peskine-arm Sep 4, 2024
ae710c8
Test dtls_client
gilles-peskine-arm Sep 4, 2024
a21e893
Test ssl_server
gilles-peskine-arm Sep 4, 2024
3abca95
Test ssl_pthread_server
gilles-peskine-arm Sep 4, 2024
c83e56c
Test ssl_fork_server
gilles-peskine-arm Sep 4, 2024
6b4d693
Test dtls_server
gilles-peskine-arm Sep 4, 2024
3338821
Declare OpenSSL version dependency for TLS 1.3 test cases
gilles-peskine-arm Sep 4, 2024
8db2b79
Declare GnuTLS version dependency for TLS 1.3 test cases
gilles-peskine-arm Sep 5, 2024
6ef5239
Compatibiliy with older OpenSSL and GnuTLS
gilles-peskine-arm Sep 4, 2024
530cb41
ssl_server: Allow the client to close the connection first
gilles-peskine-arm Sep 4, 2024
6e3de21
dtls_client: don't force the use of IPv6
gilles-peskine-arm Sep 5, 2024
c3d1a1d
Test SSL sample programs against each other and ssl_client2, ssl_server2
gilles-peskine-arm Sep 9, 2024
cc7d6ae
Note known issue about test cases skipped in TLS 1.3-only builds
gilles-peskine-arm Sep 11, 2024
e13ff09
Use OPENSSL_NEXT for a test case that uses IPv6 when available
gilles-peskine-arm Sep 13, 2024
f9ad830
Skip ssl_server in config-suite-b
gilles-peskine-arm Sep 13, 2024
5333425
Always call psa_crypto_init before using TLS
gilles-peskine-arm Sep 25, 2024
465837b
Disable session tickets for ssl_client1 when using TLS 1.3
gilles-peskine-arm Sep 25, 2024
3c16e99
Update framework to the main branch
gilles-peskine-arm Sep 26, 2024
f88f6d6
Stop testing without PSA
gilles-peskine-arm Sep 26, 2024
5602651
Merge pull request #9604 from gilles-peskine-arm/config-full-booleans…
mpg Sep 26, 2024
3b71262
Remove MBEDTLS_OID_C guard from static functions
Harry-Ramsey Sep 26, 2024
b268d27
Merge pull request #9413 from sezrab/split_numeric_string_conversions…
davidhorstmann-arm Sep 26, 2024
e4669a5
Remove legacy symbol definitions pt 1
eleuzi01 Aug 20, 2024
404311a
Remove MBEDTLS_ECP_HAVE_SECP256R1 leftovers
eleuzi01 Aug 20, 2024
c0d6943
Replace legacy symbols in config files
eleuzi01 Aug 20, 2024
3b84b2e
Remove legacy symbol definitions pt 2
eleuzi01 Aug 20, 2024
05fe6e4
Remove MBEDTLS_MD_CAN_SHA512 from header files
eleuzi01 Sep 3, 2024
feb105c
Address review comment
eleuzi01 Sep 5, 2024
7f85f1f
Cosmetic changes
eleuzi01 Sep 5, 2024
9b0bdd0
Remove MBEDTLS_ECP_HAVE_xxx and MBEDTLS_MD_CAN_xxx
eleuzi01 Sep 6, 2024
bf90fa9
Update submodule to the head of framework PR
eleuzi01 Sep 26, 2024
1a09caa
Merge pull request #9638 from gilles-peskine-arm/ssl-opt-sample-progr…
davidhorstmann-arm Sep 26, 2024
0930b33
Don't use the "allow list" terminology any longer
gilles-peskine-arm Sep 26, 2024
4677129
Merge pull request #9558 from gilles-peskine-arm/run-test-suites-on-s…
gilles-peskine-arm Sep 28, 2024
780e856
Update submodule
eleuzi01 Sep 30, 2024
96c6f5e
Merge pull request #9492 from eleuzi01/remove-legacy-symbol-definitions
ronald-cron-arm Oct 1, 2024
713127d
dtls_server: allow unexpected messages during handshake
gilles-peskine-arm Oct 1, 2024
13b4954
Remove "error" allowance in dtls_server
gilles-peskine-arm Oct 1, 2024
cb42f29
Merge pull request #9662 from gilles-peskine-arm/dtls_server-allow_un…
mpg Oct 2, 2024
38d4c91
Merge pull request #9594 from gilles-peskine-arm/analyze_outcomes-cla…
gilles-peskine-arm Oct 3, 2024
1e5dbd3
Create a module to split test case collection from checks
gilles-peskine-arm Oct 3, 2024
c8c83d0
Split test case collection from checks
gilles-peskine-arm Oct 3, 2024
9f930e0
Create a module to split branch-independent code out of analyze_outco…
gilles-peskine-arm Oct 3, 2024
49c77dd
Remove sample ignore list elements for coverage
gilles-peskine-arm Oct 3, 2024
9d78e87
Missing NotImplementedError in abstract method
gilles-peskine-arm Oct 3, 2024
ad02d44
Don't reuse a variable name inside a function
gilles-peskine-arm Oct 3, 2024
005dca6
Typecheck main
gilles-peskine-arm Oct 3, 2024
e41cde5
Pass KNOWN_TASKS as an argument to main
gilles-peskine-arm Oct 3, 2024
082eade
Separate code and data of outcome analysis
gilles-peskine-arm Oct 3, 2024
3146772
Adjust paths for impending moves to the framework
gilles-peskine-arm Oct 3, 2024
199c919
Upgrade mypy to the last version supporting Python 3.6
gilles-peskine-arm Sep 23, 2024
fecb9b9
Downgrade mypy to a version that works with our code base
gilles-peskine-arm Sep 23, 2024
2c17ec7
Default to allowing partial test coverage
gilles-peskine-arm Oct 4, 2024
ce00cfe
Move some files to framework repository
gilles-peskine-arm Oct 9, 2024
4103f92
Merge branch 'tmp-branch-move-files-to-framework' into move-files-int…
gilles-peskine-arm Oct 9, 2024
48c005f
Adjust import path after script moves
gilles-peskine-arm Oct 3, 2024
67e415f
Also write to a log file by default
gilles-peskine-arm Oct 3, 2024
b111d9f
Support compressed outcome files transparently
gilles-peskine-arm Oct 3, 2024
9c9d9fa
Default to requiring full test coverage
gilles-peskine-arm Oct 4, 2024
d63709d
collect_test_cases.py is now a separate module
gilles-peskine-arm Oct 8, 2024
6759e80
Make open calls more uniform
gilles-peskine-arm Oct 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions scripts/check_test_cases.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
#!/usr/bin/env python3

"""Sanity checks for test data."""

# Copyright The Mbed TLS Contributors
# SPDX-License-Identifier: Apache-2.0 OR GPL-2.0-or-later

import argparse
import re
import sys

from mbedtls_framework import collect_test_cases


class DescriptionChecker(collect_test_cases.TestDescriptionExplorer):
"""Check all test case descriptions.

* Check that each description is valid (length, allowed character set, etc.).
* Check that there is no duplicated description inside of one test suite.
"""

def __init__(self, results):
self.results = results

def new_per_file_state(self):
"""Dictionary mapping descriptions to their line number."""
return {}

def process_test_case(self, per_file_state,
file_name, line_number, description):
"""Check test case descriptions for errors."""
results = self.results
seen = per_file_state
if description in seen:
results.error(file_name, line_number,
'Duplicate description (also line {})',
seen[description])
return
if re.search(br'[\t;]', description):
results.error(file_name, line_number,
'Forbidden character \'{}\' in description',
re.search(br'[\t;]', description).group(0).decode('ascii'))
if re.search(br'[^ -~]', description):
results.error(file_name, line_number,
'Non-ASCII character in description')
if len(description) > 66:
results.warning(file_name, line_number,
'Test description too long ({} > 66)',
len(description))
seen[description] = line_number

def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--list-all',
action='store_true',
help='List all test cases, without doing checks')
parser.add_argument('--quiet', '-q',
action='store_true',
help='Hide warnings')
parser.add_argument('--verbose', '-v',
action='store_false', dest='quiet',
help='Show warnings (default: on; undoes --quiet)')
options = parser.parse_args()
if options.list_all:
descriptions = collect_test_cases.collect_available_test_cases()
sys.stdout.write('\n'.join(descriptions + ['']))
return
results = collect_test_cases.Results(options)
checker = DescriptionChecker(results)
try:
checker.walk_all()
except collect_test_cases.ScriptOutputError as e:
results.error(e.script_name, e.idx,
'"{}" should be listed as "<suite_name>;<description>"',
e.line)
if (results.warnings or results.errors) and not options.quiet:
sys.stderr.write('{}: {} errors, {} warnings\n'
.format(sys.argv[0], results.errors, results.warnings))
sys.exit(1 if results.errors else 0)

if __name__ == '__main__':
main()
166 changes: 166 additions & 0 deletions scripts/mbedtls_framework/collect_test_cases.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
"""Discover all the test cases (unit tests and SSL tests)."""

# Copyright The Mbed TLS Contributors
# SPDX-License-Identifier: Apache-2.0 OR GPL-2.0-or-later

import glob
import os
import re
import subprocess
import sys

from . import build_tree


class ScriptOutputError(ValueError):
"""A kind of ValueError that indicates we found
the script doesn't list test cases in an expected
pattern.
"""

@property
def script_name(self):
return super().args[0]

@property
def idx(self):
return super().args[1]

@property
def line(self):
return super().args[2]

class Results:
"""Store file and line information about errors or warnings in test suites."""

def __init__(self, options):
self.errors = 0
self.warnings = 0
self.ignore_warnings = options.quiet

def error(self, file_name, line_number, fmt, *args):
sys.stderr.write(('{}:{}:ERROR:' + fmt + '\n').
format(file_name, line_number, *args))
self.errors += 1

def warning(self, file_name, line_number, fmt, *args):
if not self.ignore_warnings:
sys.stderr.write(('{}:{}:Warning:' + fmt + '\n')
.format(file_name, line_number, *args))
self.warnings += 1

class TestDescriptionExplorer:
"""An iterator over test cases with descriptions.

The test cases that have descriptions are:
* Individual unit tests (entries in a .data file) in test suites.
* Individual test cases in ssl-opt.sh.

This is an abstract class. To use it, derive a class that implements
the process_test_case method, and call walk_all().
"""

def process_test_case(self, per_file_state,
file_name, line_number, description):
"""Process a test case.

per_file_state: an object created by new_per_file_state() at the beginning
of each file.
file_name: a relative path to the file containing the test case.
line_number: the line number in the given file.
description: the test case description as a byte string.
"""
raise NotImplementedError

def new_per_file_state(self):
"""Return a new per-file state object.

The default per-file state object is None. Child classes that require per-file
state may override this method.
"""
#pylint: disable=no-self-use
return None

def walk_test_suite(self, data_file_name):
"""Iterate over the test cases in the given unit test data file."""
in_paragraph = False
descriptions = self.new_per_file_state() # pylint: disable=assignment-from-none
with open(data_file_name, 'rb') as data_file:
for line_number, line in enumerate(data_file, 1):
line = line.rstrip(b'\r\n')
if not line:
in_paragraph = False
continue
if line.startswith(b'#'):
continue
if not in_paragraph:
# This is a test case description line.
self.process_test_case(descriptions,
data_file_name, line_number, line)
in_paragraph = True

def collect_from_script(self, script_name):
"""Collect the test cases in a script by calling its listing test cases
option"""
descriptions = self.new_per_file_state() # pylint: disable=assignment-from-none
listed = subprocess.check_output(['sh', script_name, '--list-test-cases'])
# Assume test file is responsible for printing identical format of
# test case description between --list-test-cases and its OUTCOME.CSV
#
# idx indicates the number of test case since there is no line number
# in the script for each test case.
for idx, line in enumerate(listed.splitlines()):
# We are expecting the script to list the test cases in
# `<suite_name>;<description>` pattern.
script_outputs = line.split(b';', 1)
if len(script_outputs) == 2:
suite_name, description = script_outputs
else:
raise ScriptOutputError(script_name, idx, line.decode("utf-8"))

self.process_test_case(descriptions,
suite_name.decode('utf-8'),
idx,
description.rstrip())

@staticmethod
def collect_test_directories():
"""Get the relative path for the TLS and Crypto test directories."""
mbedtls_root = build_tree.guess_mbedtls_root()
directories = [os.path.join(mbedtls_root, 'tests'),
os.path.join(mbedtls_root, 'tf-psa-crypto', 'tests')]
directories = [os.path.relpath(p) for p in directories]
return directories

def walk_all(self):
"""Iterate over all named test cases."""
test_directories = self.collect_test_directories()
for directory in test_directories:
for data_file_name in glob.glob(os.path.join(directory, 'suites',
'*.data')):
self.walk_test_suite(data_file_name)

for sh_file in ['ssl-opt.sh', 'compat.sh']:
sh_file = os.path.join(directory, sh_file)
if os.path.isfile(sh_file):
self.collect_from_script(sh_file)

class TestDescriptions(TestDescriptionExplorer):
"""Collect the available test cases."""

def __init__(self):
super().__init__()
self.descriptions = set()

def process_test_case(self, _per_file_state,
file_name, _line_number, description):
"""Record an available test case."""
base_name = re.sub(r'\.[^.]*$', '', re.sub(r'.*/', '', file_name))
key = ';'.join([base_name, description.decode('utf-8')])
self.descriptions.add(key)

def collect_available_test_cases():
"""Collect the available test cases."""
explorer = TestDescriptions()
explorer.walk_all()
return sorted(explorer.descriptions)
Loading