crypto: make malloc failure check cross-platform

Checklist

• make -j4 test (UNIX), or vcbuild test nosign (Windows) passes
• commit message follows commit guidelines

Affected core subsystem(s)

crypto

Description of change

malloc(0) may return NULL on some platforms. Do not report out-of-memory error unless malloc was passed a number greater than 0.

Marking as in progress for the moment. This is an attempt to fix some of the perma-flaky tests on AIX.