test: validate arguments to mustCall(), fix incorrect tests

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• commit message follows commit guidelines

Affected core subsystem(s)

tests

Notes: this is a subset of the changes in #9031 as requested by @mscdex and @targos.