I've got a simple test to determine credibility in an argument. Simply listen to what someone is not saying, or look at what they're not doing, in defense of their POV. The assumption is that someone really wants to win an argument, and will do what it takes. If they're not doing so, you have to wonder why.
In this case, the other billionaires are NOT showing how Buffett is wrong. Logically, wouldn't that be the thing to do? If he's full of it, then just present a simple calculation or some simple logic to show why he's out to lunch. Game over. Problem solved. So if they're not, either A) They're morons who didn't think of this, or B) They have the proof but left it in their other pants, or C) he's right.
I'm quite willing to listen to arguments for why I'm wrong. OK, go!