I found a way to totally hide the buttons:
function MakeButtonVisible(buttonValue, isVisible)
var button = RIBBON_BAR.findButtonByValue(buttonValue);
button.get_element().style.display = isVisible ? "" : "none";
That being said, I encounter other issues. For example the width of the Group remains the same as though all buttons are visible - funny thing is it works fine in compatibility mode in IE.
Also if you have a very wide group (with many large buttons) which fits fine in the screen and you switch tabs, when you get to the tab with the wide group all buttons are converted to small buttons.
I am not sure all those issues are related, but they are annoying...