I am not familiar with the functionality of the dnnModal control, yet you can work with the RadWindow by using its client-side API (http://www.telerik.com/help/aspnet-ajax/window-programming-radwindow-methods.html
) - get a reference to it and call its reload() method, for example. Or set a new url via the setUrl() method. Or call a function in the content page as explained here
if you need some work done there.
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed