For historical reasons, the "ignored" tests in outcome analysis are not
actually ignored: they must not be covered, otherwise the script complains
about an unnecessary exception. In coverage analysis, rename this behavior
to "uncovered", and have "ignored" tests be actually ignored. In driver test
parity analysis, which is now only done in the 3.6 LTS branch, keep the
historical behavior
Consuming branches are currently defining `IGNORED_TESTS` with the
expectation that the test cases must be uncovered. They will need to rename
their definition to `UNCOVERED_TESTS`.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Outcome analysis tasks can have "ignored" tests. Both coverage and driver
tasks actually don't ignore "ignored" tests: an "ignored" test must fail the
verification if it wasn't ignored.
In preparation for distinguishing between truly ignored tests and tests that
must be uncovered, generalize the test case lookup mechanism.
No intended behavior change for `CoverageTask` and `DriverVSReference`.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Turn functions that generate test cases into methods of the generator class.
Functions that used to take an API argument are now implemented in the
generic class, and API-specific functions are now methods of the
corresponding API-specific concrete class.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The `read_file_lines` context manager supported either text or binary
streams, based on a parameter passed to the constructor. But the type
annotation on the iterator claimed that all lines were text. The version of
mypy that we use on the CI was happy with that, but modern versions are not.
The advantage of `read_file_lines` over built-in functions is better
tracking of line numbers. We never took advantage of this in our code with
binary streams. Change the one place where `read_file_lines` was used with a
binary stream to use built-in functions instead, and specialize
`read_file_lines` to text streams.
This fixes a legitimate complaint of modern mypy on `macro_collectory.py`.
There was no runtime bug: the behavior was correct, only the type
annotations were wrong.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
When building Mbed TLS with test drivers, "psa_crypto_mldsa.h" is not on the
include path. Rather than get it on, which seems complicated and is not
desirable in the long term, arrange to do without this header. We just need
to define the macro PSA_ALG_IS_ML_DSA, and its behavior is defined by the
PSA Crypto API specification so pretty much set in stone.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
When TF-PSA-Crypto has pure ML-DSA, we need to handle it in the test driver
for signatures. But we must not try to reference ML-DSA identifiers in
TF-PSA-Crypto branches where they don't exist yet, even though the
compilation option already exists (which notably includes the TF-PSA-Crypto
1.1.0 release).
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
We don't support ML-DSA in libtestdriver1 yet, because it's a copy of the
`builtin` driver but ML-DSA is provided by the `pqcp` driver. This means
that we can't test “driver-only” ML-DSA builds, but it should be possible to
enable ML-DSA in a build that dispatches through the test driver. This is
currently impossible because pure ML-DSA is not a sign-the-hash algorithm,
but the code in the test driver for signatures assumes that all signature
algorithms are sign-the-hash. Fix this in a minimal way by making the test
driver activate the fallback mechanism of driver dispatch when the algorithm
is pure ML-DSA. (Don't do this for all algorithms that are not sign-the-hash,
because in general, we do want the test driver to fail if it's given an
algorithm that it doesn't support.)
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
When running tests that use mbedtls_test_fork_run_child() under Valgrind,
bad things happen. Specifically:
* Valgrind reports leaks in the child. Those leaks do exist, but they're
deliberate since we avoid cleaning up in the child (because we want to
do as little as possible in the child, and because cleanups such as
destroying PSA persistent keys would have undesirable effects outside
the child process).
* Valgrind's overridden `_exit()` doesn't just perform checks, but also for
some reason it causes the file description on the .datax file to seek
backwards, causing tests to run again in a loop.
Avoid this by calling `execve()` (via `execlp()`) rather than `_exit()` if
it looks like the test is run under Valgrind. This is safe as long as
Valgrind isn't run with `--trace-children=yes`.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
If we rely on `tf_psa_crypto_common.h`, it's too late. And `common.h in 3.6
doesn't have platform requirements.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Partially revert "Give test code access to internal macros of the library".
I think that it would be better not to require every user of `test/macros.h`
to have access to the library source, but it's out of scope here.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The child code isn't supposed to cause memory corruption, but if it does,
try to report a problem rather than mess up further.
Adapt the code to report the failure to the parent accordingly. In
particular, we need to make sure that the first byte written to the
reporting pipe is the result code in all cases, so don't jump over the
writing of the result code.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
In the success case, we were only reporting the correct data written by the
child if the data was read in a single `read` call.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
When reading data from the child, if the child reports a failure, the parent
expects the child to write an `mbedtls_test_info_t` structure, no less, no
more. To achieve this, we try reading at least one byte more, and check that
we couldn't read more than the expected size. This commit fixes two bugs:
* On success, don't require the child to fill the output buffer. This check
was only intended for the failure case, but was accidentally put in the
wrong place.
* On failure, we weren't checking that the child had written at least the
expected size, which could have been worse (we'd end up with a
child_test_info structure that's only partially initialized).
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Run some code in a child process. Propagate output from the child if the
test succeeds, and propagate the test result information otherwise.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Include `"test_common.h"` as the first thing in C files, and
`<test/build_info.h>` as the first thing in header files.
This requires moving `<test/test_common.h>` to its intended location
`"test_common.h"`.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The header `test_common.h` contains two kinds of things:
* Things to do at the beginning of individual C files. Specifically,
defining macros that notify system headers about what we want from them.
Keep those in `test_common.h`, which will subsequently be moved out of
the include directory.
* Things to do at the beginning of every header. In particular, read the
library configuration. Move them to a new header `build_info.h`,
which is the only one intended to be included from headers.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
We were missing `mbedtls_framework/code_wrapper/*.py` because we were not
traversing the directory recursively.
Also improve the comment explaining the duplicate-code workaround for code
that's being moved between files in different repositories.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Ben Taylor