For historical reasons, the "ignored" tests in outcome analysis are not
actually ignored: they must not be covered, otherwise the script complains
about an unnecessary exception. In coverage analysis, rename this behavior
to "uncovered", and have "ignored" tests be actually ignored. In driver test
parity analysis, which is now only done in the 3.6 LTS branch, keep the
historical behavior
Consuming branches are currently defining `IGNORED_TESTS` with the
expectation that the test cases must be uncovered. They will need to rename
their definition to `UNCOVERED_TESTS`.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Outcome analysis tasks can have "ignored" tests. Both coverage and driver
tasks actually don't ignore "ignored" tests: an "ignored" test must fail the
verification if it wasn't ignored.
In preparation for distinguishing between truly ignored tests and tests that
must be uncovered, generalize the test case lookup mechanism.
No intended behavior change for `CoverageTask` and `DriverVSReference`.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
When running tests that use mbedtls_test_fork_run_child() under Valgrind,
bad things happen. Specifically:
* Valgrind reports leaks in the child. Those leaks do exist, but they're
deliberate since we avoid cleaning up in the child (because we want to
do as little as possible in the child, and because cleanups such as
destroying PSA persistent keys would have undesirable effects outside
the child process).
* Valgrind's overridden `_exit()` doesn't just perform checks, but also for
some reason it causes the file description on the .datax file to seek
backwards, causing tests to run again in a loop.
Avoid this by calling `execve()` (via `execlp()`) rather than `_exit()` if
it looks like the test is run under Valgrind. This is safe as long as
Valgrind isn't run with `--trace-children=yes`.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
If we rely on `tf_psa_crypto_common.h`, it's too late. And `common.h in 3.6
doesn't have platform requirements.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Partially revert "Give test code access to internal macros of the library".
I think that it would be better not to require every user of `test/macros.h`
to have access to the library source, but it's out of scope here.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The child code isn't supposed to cause memory corruption, but if it does,
try to report a problem rather than mess up further.
Adapt the code to report the failure to the parent accordingly. In
particular, we need to make sure that the first byte written to the
reporting pipe is the result code in all cases, so don't jump over the
writing of the result code.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
In the success case, we were only reporting the correct data written by the
child if the data was read in a single `read` call.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
When reading data from the child, if the child reports a failure, the parent
expects the child to write an `mbedtls_test_info_t` structure, no less, no
more. To achieve this, we try reading at least one byte more, and check that
we couldn't read more than the expected size. This commit fixes two bugs:
* On success, don't require the child to fill the output buffer. This check
was only intended for the failure case, but was accidentally put in the
wrong place.
* On failure, we weren't checking that the child had written at least the
expected size, which could have been worse (we'd end up with a
child_test_info structure that's only partially initialized).
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Run some code in a child process. Propagate output from the child if the
test succeeds, and propagate the test result information otherwise.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Include `"test_common.h"` as the first thing in C files, and
`<test/build_info.h>` as the first thing in header files.
This requires moving `<test/test_common.h>` to its intended location
`"test_common.h"`.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The header `test_common.h` contains two kinds of things:
* Things to do at the beginning of individual C files. Specifically,
defining macros that notify system headers about what we want from them.
Keep those in `test_common.h`, which will subsequently be moved out of
the include directory.
* Things to do at the beginning of every header. In particular, read the
library configuration. Move them to a new header `build_info.h`,
which is the only one intended to be included from headers.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
We were missing `mbedtls_framework/code_wrapper/*.py` because we were not
traversing the directory recursively.
Also improve the comment explaining the duplicate-code workaround for code
that's being moved between files in different repositories.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Move a bunch of files from `scripts` and `mbedtls/scripts` to the framework.
The following files will be added (moved from Mbed TLS `development`):
* `scripts/ecp_comb_table.py`
* `scripts/massif_max.pl`
* `tests/scripts/audit-validity-dates.py` (moved to `scripts/`)
* `tests/scripts/gen_ctr_drbg.pl` (moved to `scripts/`)
* `tests/scripts/gen_gcm_decrypt.pl` (moved to `scripts/`)
* `tests/scripts/gen_gcm_encrypt.pl` (moved to `scripts/`)
* `tests/scripts/gen_pkcs1_v21_sign_verify.pl` (moved to `scripts/`)
* `tests/scripts/generate-afl-tests.sh` (moved to `scripts/`)
* `tests/scripts/generate_server9_bad_saltlen.py` (moved to `scripts/`)
* `tests/scripts/run-metatests.sh` (moved to `scripts/`)
* `tests/scripts/run_demos.py` (moved to `scripts/`)
* `tests/scripts/test_config_script.py` (moved to `scripts/`)
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Tests using randomized signatures were currently randomized. We need the
output of the script to be reproducible, so this won't do.
Force randomized signatures to use a specific RNG which is reset before
constructing each test case. This way, the script's output is reproducible.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This lets us define things that we want to have everywhere in test code. In
particular, this lets us define platform-specific symbols that influence
what system headers declare. This also takes care of including the library
configuration.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Gilles Peskine