I am not aware of another problem that the RadColorPicker control has when used with the Modal Popup Extender except the problem with incorrect size of its elements, which is a general problem (manifests when the RadColorPicker is rendered in a parent with display:none)
rather than something specific to the Modal Popup Extender.
As you do not mention which version and which browser you use, I tested your code with version 2009.1 402 under IE7, IE8 and FF3 and the only problem I was able to reproduce is not one of the problems you describe - the width of the RadSliders in the RGBSliders tab is not correct. This is something that you can reproduce even with ShowIcon=true and it is a problem in the code of the RadColorPicker control which we will fix it. However, I was not able to reproduce the problem with the preview box not refreshing while you hover over the color boxes in the WebPalette view. And, when I changed the value of a RGB slider, the preview updated as expected as well and when I clicked the Apply button, the currently previewed color was set as selected color, which is again expected.
Could you please prepare and send me a simple running project, demonstrating the problems you have as well as detailed information on each problem - e.g. steps to reproduce, result you get and expected result?
Basically, when you have ShowIcon=true the palette of the control is created on the client, whereas when you have ShowIcon=false, the palette is created on the server. However, this should not influence the functionality of the control.
the Telerik team