I hope you can perform this by attaching 'OnClientItemClicking'
client event to RadMenu. In the event handler check for the Item text and cancel the event by using args.set_cancel(true) method. Now you can easily show the window in a moda window from client code.
Hope this suggestion helps,