The issue there is that the solution to untrusted hardware is to apply a knowledge-hiding transform. So far, all of the ones proposed here have been based on shared secrets.
With your password solution, it hides a coercion signal.
With Norswap's solution, it prevents redirection by hiding which choice was made. (In the bank calculator variant, it just encapsulates the shared secret in the device).
The issue there is that arbitrarily saying "And what if _that_ is infected" results in an infinite regression. None of the computationally-secure knowledge-hiding transforms are simple enough to do reliably by hand without either reference material (the letter, which as you pointed out is interceptible) or a device (which basically encapsulates the reference material or its generation process).
However, there are some interesting cases if we reduce the problem space a bit by showing that the token _cannot_ be infected, and making replacement detectable.
First of all, the token really just needs to perform one operation: MAC(shared_key, text). This is incredibly simple, and if you presume a fixed-size text can be pure hardware - no risk of infection.
Second, there's actually a simple solution to the steal-and-replace problem: Prior to use, the user inputs a challenge and it must reply correctly. Supporting this is similarly simple in pure hardware - if it stores four (post-MAC) challenges and four responses, that's a small buffer of memory elements and a comparator - but a replacement device could not reply correctly. This relies on the user checking, but that's kind of an issue with any such system.
Third, make generating a value with the token take a tuple of (passphrase, value-to-hide). This passphrase then acts as the coercion signal from your solution.
Now, a token that records the password will give an incorrect challenge (alerting the user) and doesn't have the MAC key (making it useless to coerce them into using it, since they'll give it the coercion signal too, which the attacker will faithfully type into the real device).