Are you adding fresh grounds in a bag or just adding strong brewed coffee to the wort? Either way boiling water with it should kill the nasties. I found adding coffee too early gives a tobacco flavour which isn't good at all!
I haven't decided yet, but I definitely want to avoid imparting too many tannins, so I'll probably cold brew the grounds and put that in after the fermentation is complete. I've read that adding the beans straight in leaves more oil in the beer. We wound up accidentally buying some decaf Kona while we were down there, so I'll use that for the beer. I also have some cacao nibs leftover from a Mexican chocolate porter, so I may toss some of those in as well.