sha256: fold high bits of ilen into total[1] on 64-bit platforms

mbedtls_sha256_update() tracks the running byte count in a 64-bit
counter split across ctx->total[0]/[1] (32 bits each), used at
finalization to encode the message length into the padding.

ctx->total[0] += (uint32_t) ilen truncates ilen to its low 32 bits
before adding, and the overflow check right after (ctx->total[0] <
(uint32_t) ilen) only detects the single wraparound of that 32-bit
addition. Neither step ever looks at bits 32 and up of ilen itself.
So when a single update() call is passed >= 2^32 bytes on a 64-bit
platform (size_t wider than 32 bits), every byte is correctly hashed
into the message, but the length folded into the final padding block
is wrong by a multiple of 2^32 bytes, producing an incorrect digest
despite no error being returned.

Fold ilen>>32 into total[1] directly, guarded on SIZE_MAX so 32-bit
platforms (where ilen cannot exceed 2^32-1 and the shift would be
UB-adjacent) are unaffected.

Verified live: generated a 4GiB+64 byte random file, hashed it with
mbedtls_sha256_update() in a single call both before and after this
fix, and compared against sha256sum of the same file.
  - before: 803ab176fbafaf3cf345cbe22e2a35715c3ae8f7fbf241a00677d4d733db0214 (wrong)
  - after:  21a1f671cd9797227f341a243fed82c8fe95a769a58a6f3965e958f047e969d6 (matches sha256sum)
Also re-verified normal-sized inputs (empty string, "abc") against
known SHA-256 vectors, and that a two-call split of a 100-byte input
still agrees with a single whole-buffer call.

Fixes #10964.

Signed-off-by: Johnson Ajibi <[email protected]>

ChangeLog.d/sha256-update-ilen-over-4gib.txt

@@ -0,0 +1,6 @@
+Bugfix
+   * Fix mbedtls_sha256_update() computing the wrong digest when a single
+     call is passed 4GiB (2^32 bytes) or more of input on a 64-bit
+     platform: the upper 32 bits of the input length were silently
+     dropped from the running byte count used at finalization. Reported
+     by TimothyPMann in #10964.

library/sha256.c

@@ -667,6 +667,10 @@ int mbedtls_sha256_update(mbedtls_sha256_context *ctx,
         ctx->total[1]++;
     }
 
+#if SIZE_MAX > 0xFFFFFFFF
+    ctx->total[1] += (uint32_t) (ilen >> 32);
+#endif
+
     if (left && ilen >= fill) {
         memcpy((void *) (ctx->buffer + left), input, fill);