build: do not pass target toolchain flags to host toolchain
Fixes target toolchain arguments being passed to the host toolchain when cross-compiling. For example, -m64 is not available on aarch64.
See also https://github.com/NixOS/nixpkgs/pull/238377#discussion_r1245148404