In the current version of RadDocking control it is not possible to limit the number of PaneGroups in the Docking control. I would suggest to you to use the PreviewShowCompass
event of the control and in it add your custom docking logic.
I created and attached a sample project for you that shows how to achieve the desired behavior, give it a try.
Hope this is helpful.
the Telerik team