You will be able to keep your public/private key pairs on a USB stick with a reasonable level of security.
Using this script, you will create a local ssh keystore, use it to decrypt your key, and delete it when you are done.
- OS X, Linux: fully supported
- Windows: some manual intervention required