don't touch the inside surface of your goggle lens - ever.
if you see actual water droplets sitting on the inner lens you can use the corner of a paper towel to "lift" the water drops off, but never ever ever go rubbing or wiping them.
the only "exception" is to use cat crap to try and revitalize the anti-fog but at that point they're pretty much ruined anyway.
Actually some Smith goggles have a technology that according to them can be cleaned without wiping off...
Smith Technology | SmithOptics.com
"A hydrophilic chemical treatment creates a micro-etched surface on the inner lens to absorb moisture before fog can form. Because Fog-X is a treatment that is actually burned into the lens, it can’t be wiped away like other anti-fog coatings."
No idea which of their lineup comes with this treatment but it sounds promising.