Text mode ('t') is the default for the open builtin, but not for gzip.open
and its imitators. Always specify it explicitly to avoid making maintiners
wonder if there's an intended difference in behavior.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Command-line options allow choosing whether a never-executed test results in
a warning or an error. Also, a consuming script can make the default
permissive by setting FULL_COVERAGE_BY_DEFAULT to False.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Transparently read outcome files compressed with xz (which we currently use
on the CI) or with gzip.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Write the log of outcome analysis to a log file by default. This is a cheap
way of getting the outcome analysis log in an easy-to-access form on the CI:
with our current CI scripts, they are now automatically available as an
artifact called `analyze_outcomes.log`.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Currently, many test cases are not executed. A follow-up pull request will
take care of that. In the meantime, continue allowing partial test coverage.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Upgrade mypy to 0.971, which is the last version that supports Python 3.6
(the oldest Python version that we currently run on the CI).
This fixes the error
```
framework/scripts/mbedtls_framework/outcome_analysis.py:119: error: Incompatible return value type (got "IO[Any]", expected "TextIO")
framework/scripts/mbedtls_framework/outcome_analysis.py:121: error: Incompatible return value type (got "IO[Any]", expected "TextIO")
```
As far as I can tell the fix is https://github.com/python/mypy/pull/9275
which was released in mypy 0.940.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Place the code of outcome analysis (auxiliary functions, tasks, command line
entry point) into a separate module, which will be moved to the
version-independent framework repository so that it can be shared between
maintained branches. Keep the branch-specific list of driver components and
ignore lists in the per-repository script.
We keep the executable script at `tests/scripts/analyze_outcomes.py`. It's
simpler that way, because that path is hard-coded in CI scripts.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Always have tasks_list be a list, not potentially some fancier iterable.
Bypass mypy's somewhat legitimate complaint about REFERENCE and DRIVER in
task_class: they could potentially be instance attributes, but we rely on
them being class attributes. Python does normally guarantee their existence
as class attributes (unless a derived class explicitly deletes them), but
they could be overridden by an instance attribute; that's just something
we don't do, so the class attribute's value is legitimate. We can't
expect mypy to know that, so work around its complaint.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Use different names for task name, a task class and a task instance. The
interpreter doesn't care, but it's less confusing for both humans and type
checkers.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The ignore list for coverage only has two test cases out of ~10000 that are
currently reported as not executed. This is a drop in the sea and not
useful. Remove them so that the class can be used generically. A follow-up
will construct a comprehensive ignore list.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Move the test case collection code out of check_test_cases.py and into its
own module. This allows outcome analysis to depend only on the new module
and not on check_test_cases.py.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Now that dtls_server doesn't print "error" when it receives stray messages
while it's waiting for a second handshake, have the tests fail if "error" is
printed for some other reason.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
If MBEDTLS_ERR_SSL_UNEXPECTED_MESSAGE happens during the handshake, don't
show it as an "error". It might be an error, but it might also be a fact of
life if it happens during the second or more handshake: it can be a
duplicated packet or a close_notify alert from the previous connection,
which is hard to avoid and harmless.
Fixes#9652.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
And add temporary solution so it works with 3.6 and 4.0
Temporary solution to be resolved in #51.
Signed-off-by: Elena Uziunaite <elena.uziunaite@arm.com>
This commit removes the MBEDTLS_OID_C guard from the static functions in
the library/x509_create.c as this function is no longer included in the
oid.c file.
Signed-off-by: Harry Ramsey <harry.ramsey@arm.com>
Stop testing configurations without PSA (MBEDTLS_PSA_CRYPTO_C or at least
MBEDTLS_PSA_CRYPTO_CLIENT). No future release from this branch will support
such configurations, and we can no longer build the SSL sample programs
without psa_crypto_init.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Gilles Peskine