This is expected as the ShowDialog method in XBAP is asynchronous - this means that it returns immediately and shows a modal window (it works like in Silverlight). In WPF desktop application this method is synchronous - this behavior comes from the standard Window control.
The window that is shown when you call the ShowDialog method is still modal, no matter it is shown asynchronously.
Hope this information helps.
the Telerik team