Num Lock breaks everything
The game considers Num Lock (or Caps and maybe even scroll lock) + any key to be a different key than that key without Numlock on.
I've decided that the game should simply ignore Numlock entirely. It should also ignore capslock status for non-alphabetical keys.