Fenrir fixes

[cconlon]

This PR resolves a batch of 13 findings from Fenrir. Changes are hardening, input validation, consistency, and thread-safety improvements. No public API changes.

Changes

Public key validation on key agreement and import

• F-4359: validate peer DH public key in WolfCryptKeyAgreement.engineDoPhase()
• F-4855: validate peer ECC public key in WolfCryptKeyAgreement.engineDoPhase()
• F-5905: enforce 1 < Y < p-1 range on DH public key in the KeyFactory import path
• F-5906: enforce 1 < Y < p-1 range on DH public key in the X.509 import path

Consistent exception and unpadding behavior

• F-5030: constant-time PKCS#7 unpadding with a uniform BadPaddingException
• F-5066: uniform exception for RSA PKCS#1 v1.5 unwrap failures

Key and IV reuse handling

• F-5029: reject AES-GCM key and IV reuse for encryption in WolfCryptCipher
• F-5222: reject AES-GMAC key and IV reuse in WolfCryptMac

Thread-safety and object lifecycle

• F-4366: fix use-after-free race in CertManager nativeVerifyCallback
• F-5794: synchronize FIPS error callback access in jni_fips.c
• F-5887: close a Curve25519 release split-lock race by freeing under one lock

Certificate path and revocation handling

• F-5795: match qualified jdk.certpath.disabledAlgorithms entries by algorithm name
• F-5888: hard-fail OCSP revoked status regardless of SOFT_FAIL

[Copilot]

Pull request overview

This PR addresses a batch of Fenrir findings by hardening crypto behavior (key validation, padding/unwrapping uniformity, nonce/IV reuse rejection), and improving thread-safety in JNI callbacks, with new/updated unit tests to cover the fixes.

Changes:

• Add DH/ECC peer public key validation in key agreement and DH public-key range checks on import paths.
• Enforce uniform failure behavior for PKCS#7 unpadding and RSA PKCS#1 v1.5 unwrap, and reject AES-GCM/AES-GMAC nonce reuse for encryption/MAC operations.
• Fix JNI thread-safety issues around verify/FIPS callbacks and strengthen certpath disabled-algorithm parsing + OCSP revoked handling (plus tests).

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/test/java/com/wolfssl/wolfcrypt/test/WolfSSLCertManagerVerifyCallbackTest.java	Adds concurrency-focused tests for CertManager verify callback registration/clearing.
src/test/java/com/wolfssl/wolfcrypt/test/WolfCryptTestSuite.java	Registers the new CertManager verify-callback test in the suite.
src/test/java/com/wolfssl/provider/jce/test/WolfCryptUtilTest.java	Adds regression coverage for qualified jdk.certpath.disabledAlgorithms entries.
src/test/java/com/wolfssl/provider/jce/test/WolfCryptPKIXRevocationCheckerTest.java	Adds test ensuring OCSP revoked hard-fails even under SOFT_FAIL.
src/test/java/com/wolfssl/provider/jce/test/WolfCryptMacTest.java	Updates GMAC reset behavior test and adds IV-reuse rejection coverage.
src/test/java/com/wolfssl/provider/jce/test/WolfCryptKeyAgreementTest.java	Adds negative tests for DH/ECDH invalid peer public key rejection.
src/test/java/com/wolfssl/provider/jce/test/WolfCryptDHKeyFactoryTest.java	Adds tests for DH public-key range validation in X.509 and DHPublicKeySpec paths.
src/test/java/com/wolfssl/provider/jce/test/WolfCryptCipherTest.java	Adds tests for RSA unwrap uniform failures, CBC padding message uniformity, and GCM nonce-reuse rejection.
src/main/java/com/wolfssl/wolfcrypt/Dh.java	Adds checkPublicKey() wrapper over native wc_DhCheckPubKey().
src/main/java/com/wolfssl/wolfcrypt/Curve25519.java	Adjusts release locking to avoid split-lock race during native free.
src/main/java/com/wolfssl/wolfcrypt/BlockCipher.java	Implements constant-time PKCS#7 unpadding with uniform error behavior.
src/main/java/com/wolfssl/provider/jce/WolfCryptUtil.java	Improves parsing/matching of qualified disabled-algorithm entries; ignores keySize constraints here.
src/main/java/com/wolfssl/provider/jce/WolfCryptPKIXRevocationChecker.java	Treats OCSP revoked as a hard failure with BasicReason.REVOKED regardless of SOFT_FAIL.
src/main/java/com/wolfssl/provider/jce/WolfCryptMac.java	Rejects AES-GMAC operations that would reuse key+IV without re-init.
src/main/java/com/wolfssl/provider/jce/WolfCryptKeyAgreement.java	Validates DH/ECC peer public keys before use in engineDoPhase().
src/main/java/com/wolfssl/provider/jce/WolfCryptDHPublicKey.java	Adds DH public-key Y range validation during X.509 parsing.
src/main/java/com/wolfssl/provider/jce/WolfCryptDHKeyFactory.java	Adds DH public-key Y range validation for DHPublicKeySpec generation.
src/main/java/com/wolfssl/provider/jce/WolfCryptCipher.java	Adds AES-GCM nonce reuse rejection; makes unwrap/padding failures less distinguishable.
jni/jni_wolfssl_cert_manager.c	Fixes verify-callback use-after-free by taking a local ref under mutex.
jni/jni_native_struct.c	Initializes FIPS callback mutex during JNI_OnLoad.
jni/jni_fips.c	Adds mutex-protected storage/use of FIPS error callback global ref.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.

[cconlon]

Retest this please Jenkins