The KeepInScreenBounds property of the RadColorPicker refers to the bounds of the browser's window, not another containing element. By suggestion will be to set a CSS rule to the jQuery dialog, so that the color picker doesn't go out of it, i.e. overflow: hidden.
Hope this helps.
the Telerik team