crypto: fix RSA-PSS default saltLength
Based on my understanding of RFC 8017, when hashAlgorithm
is set but saltLength
is not, the value of saltLength
associated with the key pair should default to the digest size of hashAlgorithm
, not to 0
.
I am not sure why OpenSSL uses 0
. I suspect it is because we don't call EVP_PKEY_CTX_set_rsa_pss_keygen_saltlen
and 0
is the least restrictive value, at least within OpenSSL. This behavior can still be restored by explicitly setting saltLength
to 0
.
I'd argue that this is a bug fix. If we are concerned about semverity, I could modify this PR to only affect the new options (#39927) and keep the behavior of the old options intact. Personally, I don't think it's necessary to go that route.