Go ahead - just use the goggle wipe on it when it's dry and be gentle. The only time you can't wipe/touch the inside is when it's wet. If it's dry it's fine - and needs to be done otherwise as you know, you'll have smudges and spots from where water dried.
Shit, the way some of you are talking I could be mistaken that I'm going to loose my fucking finger by putting it near the inside of my goggles.
Exactly. Just make sure you don't do it when they are wet...wiping them gently when its dry with a microfibre will be fine.