Straight to your questions:
Q: can you add a button to go to the next/previous tab?
A: Yes, you can add a button and you could implement the next/prev tab navigation. I would suggest using the TitleTemplate of the panes. When a pane is selected its TitleTemplate is placed in the PaneGroup header. By default the value from the Header property of the selected pane is displayed in the title, but this could be changed using a datatemplate. Please, refer to the following help article that explains how to set a button in the header - http://www.telerik.com/help/wpf/raddocking-how-to-add-buttons-to-the-pane-headers.html. I would suggest using a custom command and passing the RadPaneGroup as a parameter. Using the group, you could get all panes, including the selected pane, and implement the next/prev pane logic via the RadPaneGroup.SelectedItem property.
Q: can the dragging a floating tab to the tab area dock it?
A: The RadDocking control doesn't support docking a floating windows to the tab area. This sounds like a good feature request and I logged it in our PITS. You can vote for it and track its progress here - http://www.telerik.com/support/pits.aspx#/public/wpf/13163. With the current release we introduced a new feature for the RadDocking control that allows pane reordering in the groups - you could enable it setting the RadPaneGroup.AllowReorder property to true. You could dock a pane in the group and reorder it after that.
Q: can dragging docking a window to the left or right side split the area below the tab area, respecing the existing tab interface?
A: The RadDocking control uses a dock panel to arrange its parts. The RadPanes are placed in a groups, and the groups are placed in split containers. The RadDocking control works directly with the split containers and arrange them in the dock panel. When a pane (toolwindow) is docked, it is placed in a panegroup or a splitcontainer and the RadDocking control constricts some other splitcontainers and panegroups to make a place for the new docked item. If the pane (toolwindow) is docked in the center of the compass, it's inserted in the current panegroup as last one. This logic is by control design and more information about RadDocking compass can be found here - http://www.telerik.com/help/wpf/raddocking-features-compass.html
Please, let me know if this helps and do not hesitate to contact us if you have any additional questions. I am glad to assist you further.
All the best,
the Telerik team