RadTransitionControl is actually a ContentControl which plays animation when its Content is changed. So it cannot be used for animating RadWindow as the Window cannot be placed inside a ContentControl.
As to RadDocking - you should edit the ControlTemplate of the RadPaneGroup and replace the ContentPresenter in it with a TransitionControl. I have attached a sample project to demonstrate the approach. Note that I have used Implicit Styles
as this makes such customizations easier.
Hope this helps.
the Telerik team